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EXECUTIVE SUMMARY 



Concern about the changing "work ethic," "quality of working life," and. economic problems 
associated with productivity has implications for assessing former students' job satisfaction and 
their satisfaction with their education and training. These factors have been addressed in previous 
research but only in a limited manner. At the present time, the continued concern for lagging 
productivity and chronic youth unemployment points to the need for greater attention to these 
factors and their relationships to vocational education. 

According to the literature, evaluation efforts have traditionally been considered to be 
inadequate, particularly in regarcHo having ah impact upon program improvement. Several reasons 
have been posited to support thisitatement, buch as (1) programs often have multiple (and sometimes 
conflicting) goals that are difficult to evaluate, (2) programs are complex and dynamic, and (3) often 
traditional research designs are used inappropriately to answer the evaluation questions'. 

However, evaluation efforts are still included as an integral part of program activities as essential 
for evaluation, accountability, and program improvement. It is apparent that there is a need for 
providing evaluators and users of evaluation information with guidelines and practices for the increased 
use of evaluation information. For example, the 1976 vocational education legislation reflects the 
need for interrelating evaluation findings from Vocational Education Data Systems (VEDS), state 
and local advisory council vocational education reports, antfother evaluation efforts. 

This handbook is primarily designed for local and state vocational education evaluation personnel 
who are faced vyith any of the following problems: (1 ) lack of research expertise to conduct studies 
on training satisfaction and job satisfaction, and (2) lack of adequate resource (time and money) to 
perforfn systematic evaluations. 




aivattempt to make this handbook relevant to the needs of people in the field, selected state 
and local vocational evaluation personnel were a:ked to review the outlines and the drafts of the 
handbook with particular focus on the substantial contents and formats. Their suggestions and 
recommendations were analyzed and incorporated into the present form of the handbook. 

The handbook has eight chapters. Chapter 1 explains the rationale, development, and organiza- 
tion of the handbook. The next chapter is composed of two commissioned concept papers on the 
state of the art, problems, and issues of measuring training satisfaction and job satisfaction. Chapter 3 
discusses some basic evaluation considerations. The succeeding three chapters, 4 to 6, explain the 
following alternative data collection methodologies: mailed questionnaires, interview's, and some 
qualitative methods. Chapter 7, Data Analysis and Interpretations, exposes the reader to some 
descriptive and inferential statistics that are appropriate in analyzing and interpreting data on training 
satisfaction and job satisfaction. The last chapter, chapter 8, explains to the reader some strategies 
for data presentation and utilization. 
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CHAPTER 1 



INTRODUCTION 

/ 1 ■ ■ • ■ ■ . 

With inflation, high energy cost, and a slowing economy, it is becoming increasingly difficult 
for schools to obtain additional operation monies from both the taxpayers and government. Many 
local communities have difficulty passing referenda for increased school levies. Additionally, many 
states and local governments are experiencing budget problems. Thus, most education agencies are 
financially hard-pressed to mafhtain presen^t vocational programs and even harder-pressed to expand 
or to ppen new programs. The present financial squeeze, therefore, demands greater emphasis on 
cost effectiveness and cost benefits in the delivery of vocational education, and thus requires that 
decisjon making and planning in vocational education be more scientific— i.e., objective, systematic, 
and based on quality factual information. To base decision making on hope, good intentions, and 
doubtful information is to invite biased and ill-informed judgments, costly rnistakes, and even loss 
of taxpayers' support. 

Legislators have recognized this nationally increasing need for quality evaluation data, and it 
was partly this concern that resulted in the Education Amendments of 1976. Among other things, . 
this Act requires the state board, during the five-year period pf the state plan, to evaluate ih quanti- , 
tatiVe terms the effectiveness of each formally organized program or project supported by federal, 
state, and local funds.*These evaluations shall be in terms of the following: h 

a. Planning and operational processes, such as quality and availability of 
instructional offerings - >. 

b. Results of student achievement : 

c. Results of student employment success (sec. 104-402, Rules & Regulations 
. of P. 1. 93-^1 12 as amended) 

This handbook takes the view that perceptions of former vocational students re'garding their voca- 
tional training and jobs provide valuable information for accountability purposes, and for effective 
decision making in both state and local education agencies. Students who have been in programs 
provide'one of the best sources for insights and information as to the quality of vocational program 
offerings. Additionally, some educators view job satisfaction as one outcome of vocational education 
that needs consideration in determining student employment success. 

Why Is the Handbook Developed? 

This handbook is primarily designed for local and state vocational education evaluation personne 
who are faced with any of the following problems: Tl ) liniited expertise in evaluation and research 
methods, and (2) inadequate resources to conduct systematic evaluation studies. Despite these limita- 
tions, local and state education agencies must recognize the need for valid and reliable evaluation data. 
The fact, however, is that many local and state- vocational education agenjcies are presently measuring 



the training and job satisfaction, of forrner vocational students as part of their follow-up studies 
(O'Reilly and Ashe IQTS; Gray eraM978; Mertens et al. 1980). The problem is that many of these 
studies are fraught witr- serious methodological flaws, making the efforts less than useful (Mertens 
et al. 1980). 

• ' \ 

How Was the Handbook Developed? 

Th? major thrust in the development of this handbook was to produce a reference that is of 
practical importance to evaluation personnel-one that should help them solve their common 
problems in measuring the training satisfaction and job satisfaction of former vocational students 
With this objective in mind, an effort was made to conduct a systematic survey of the literature to 
get Ideas from leading practitioners in the field, and to review exemplary systems and instruments 
Concept papers on the state of the art including problems and issues were commissioned from recoq- 
Jhl^^ a f practitioners. These papers helped to develop the theoretical framework for 
this handbook In addition, the experiences of the project staff and the Evaluation arid Policy Division 
ot the National Center have served as a foundation for the materials contained in chapters 3 4 6 
-/, and 8. 

In an attempt to make this handbook relevant to the needs of practitioners, selected state and 
local vocational evaluation personnel across the country were asked to review the outlines and the 
drafts of the handbook with particular focus on the substantive contents. Their suggestions and 
recommendations were analyzed and incorporated in the present form of the handbook. 

How is the Handbook Organized? 

This handbook addresses two distinct activities: measuring training satisfaction and measuring 
job satis.action. Two major factors influenced this decision to combine the two activities into one 
handbook. First, as mentioned earliei^, stdte and local education agencies generally address both 
traming and job satisfaction in their former student follow-up studies. Second, the same methodo- 
logical principles and practices for measuring training satisfaction also apply for measuring job 
satisfaction. ! . : 

The text is divided into chapters. Briefly, chapter 1 finesents the rationale of the handbook while 
chapter 2 explains the concept papers on the state of the art, problems, and issues of measuring ' 
training satisfaction and job satisfaction. The next three chapters, 3 to 6, discuss some basic evaluation 
considerations and alternative ddta collection methodologies appropriate for measuring training and 
job satisfaction of former vocational students. Every chapter is designed to enable the readers to 
Identify the advantages and disadvantages oi each method, to understand the mechanisms of the 
different methodologies, and to determine the appropriate methodology for a particular condition 
Chapter 7 deals with data analysis and interpretation; chapter 8 discusses data presentation and 
utilization. Like the previous four chapters, exercisesand checklists^are provided, whenever appro- 
priate, to,give the reader a simulated field experience; and to the extent possible/each section has 
been written to provide the reader sufficient background/and information to understand the concejsts 
being presented. Here too, key references have been incorporated as aids for those readers who may 
be interested in pursumg a topic in greater depth. The handbook also contains an appendix of selected 
annotated bibliographic entries and references. 

Before proceeding in reading this handbook, you may find it worthwhile to assess your present 
knoyyledge^and skiljs by completing the Training Satisfaction and Job Satisfaction Profile (checklist 1 ). 
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The profile is designed to hstp you in these areas: 

• To thin . through the process of measuring the training and job satisfaction of 
former vocational students 

• To become familiar wim the issues you may not have considered 
© To gain an overview of the contents of this handbook * 

• To decide which part or section of this handbook you shoOld skim, read carefully 
or study in depth / 

i 

To complete checklist 1, rate your current level of knowledge of each question by darkeninj 
the appropriate circle in the left-hand column. The page number on the right identifies the locatio.. 
of the information in the text that is helpful in addressing the question. The following is an exampi 
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Current Level of 
Kno wIedge/Skills 
None Moderate High 



Questions 



Pages 




3. 



2. 



1. 



How should relevant decision 
makers and information users 
be identified? 



What are perceptions? 



Why should the training and 
job satisfaction of former 
yodational students be 
measured? 



10-11 



9 



36-37 



'/ 
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CHECKLIST 1: 
Training and Job SatTsfaction Profile 



None 


Current Level of 
Knowledge/Skills 

Moderate 


High 


Items 


Pages 


n 


rn 


■1 1 


1. vvny snouiQ iraininQ and job 
sa^sf action of former vocational 
students be measured? 


S> 


n 


□ 


□ 


2. What are perceptions? 


10- 11 


□ 


□ 


□ 


3. What factor^ influence training 
satisfaction? 


10 


□ 


. D 


□ 


4. What are the common problems 
in measuring training satisfaction? 


11- 20 


□ 


D 


□ 


5. What is job satisfaction? 


22 


□ 


□ 


□ 


6. What factors influence job 

SoIISTaCIlOn r 


23- 25 


□ 




□ • 


7. What are the common problems 
in measurino iob sati<;fartinn? 


Z/- 32 


□ 


□ 
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makers and information users 
be identified? 


36- 37 


□ 


D 


o 


9. How are objectives of the study 
written? 


37- 39 


□ 




□ 


10. How are respondents determined? 


39- 43 ; 


□ 




□ 


11. How are appropriate research 
aesigns lor tne stuay cnosen.*' 


* ■ -/ 

.43- 46 ; 








12. When sho\j|d a mailed 
L|UcbLionnqire ue useu; 


- t 

49- 52 / 

H 

52- 69 /: 


□ 






13. How can confidence in a data- 
collected-by-mail questionnaire 
be increased? 






□ 


14. How are questions to be asked 
determined? ' 


53- 5^/' 
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Checklist 1, (continued) 



None 


Moderate 


High 


-j r -■ 


Pages \ 




□ 


■ □ 


1 5. How are questions worded? 


56- 59 


□ 


□ 


□ 


16. How are response options 
constructed? 


59-65 


1 — I 


1 1 

■ 1 1 


1 1 

LJ 


17. Why pretest questionnaires? 


65- 67 


n 


□ 


□ 


18. How can the generalizability 
of a ojiaa-collected-by-nnail 
questionnaire be increased? 


67- 73 


□ 


n 


1 1 


19. How are questionnaires 
properly formatted? 


67- 68 


□ 


m 


1 — 1 


20. What are some strategies for 
stimulating response? 


68- 70 


n 


m 
1 1 


1 — 1 


21 . How can nonresppnse bias 
be dealt with? 


70- 73 


□ ■ 


□ 


n 


22. When should interviews 
be used? ; 


77- 78 


□ 


□ 




23. How can confidence in the ' 
data-co 1 1 ected-by- 1 ntervi e w 
be increased? 


78- 86 


n 


. — 
\ 1 


1 1 


24. How can the questionnaire 
be used in interviewing? 


80- 81 


■ □ 




□ 


25. How can probing be 
effectively conducted? - 


81- 82 


□ 






26.. How can interview bias-and 
social-desirability bias be 

minimized? 

/.( 


83- 85 


□ 


□ 


□ ! 


27. How can the generalizability 
of data collected by inter- 
view be increased? 


86- 88 


□ 


. □ 


n 

III 


vvnaL are quaiiiaiive 
evaluiation methods? 


93- 94 






□ 


29. Why use qualitative evalua- 
tion methods? 


93- 94 



ChCicklist 1 (continued) 



None 


Moderate 


High 




Pages 


□ 




1 1 


30. What is participant 
observation? 


94 


□ 


1 1 


1 — 1 
1 1 


31. Why use participant 
observation? 


95- 96 






□ 


32. What ar(|the strategic 
successful observation? 


9 


□ 
□ 


□ 

v 

Vi 


□ 
□ 


i 
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CHAPTER U 



STATE OF THE ART, PROBliEMS. AND ISSUES 



This chapter presents two concept papers on the state of the art, problems, and issues of 
measuring training satisfaction and jobj satisfaction of former vocational students. Patrick A O'Reilly 
assistant professor at the Virginia Polytechnic Institute and State University* explains the meaning of' 
perception and the liiflitations of usingj perception as a basis for determining the effectiveness qf 
vocational training programs. He arguek that student perception is influenced by a numbeC'of variables 
outiside the training process itself and that these are beyond school control. Additionally, there are 
methodological problems associated with measuring training satisfaction. 

, ■ ' ■ \' 

The second paper, by Robert S. Billings, assistant professor at The Ohio State University, present 
the meaning of job satisfaction, the factors influencing it, and the problems and issues of measuring 
It, Billings points out that the effects of vocational training on job satisfaction may. be small/due to 
the many causes of job satisfaction. For this reason, data on job satisfaction should be used with - 
caution. 



MEASURING AND USING TRAINING SATISFACTION 
OF FORMER VOCATSONAL STUDENTS 

by 

Patrick A. O'Reilly* 

It has become increasingly common, and popular, to seek input from students regarding the 
value of. educational programs in which they are or have been involved. Such input is sought when ^ 
planning as we!l as when evaluating programs, and one procedure frequently used is to conduct 
follovy-up studies. Student evaluation of teachers and teaching, as well as of curriculum and training 
experiences, has become especially common in postsecondary and four-year institutions.'One reason 
for the popularity of follow-up studies in the field of vocational education is that they are mandated; 
another reason is that follow-up can produce the kinds of product evaluation information needed by' 
program planners and evaluators. \ 

In vocational education one method frequently used to obtainNsvaluation information abdUt 
programs is to collect data from former student's regarding the degree of satisfaction that they have 
(experienced regarding their training programs. This information is most often collected as part 6f a 
follow-up effort to determine the individual's employment status. Whether or not the information 
actuaily represents students' satisfaction has not been widely investigated bv vocational educators. ' 



Assistant professor, Virginia Polytechnic Institute and State University. 
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Perceptions-V'hat They Are 



_ Some disagreement exists among psychologists regarding what a perception actually is or is not 
One group would define it as "the process of information extraction" (Fargus 1966 p 1) Or perhaps 
It would be more exactly defined as "a process of organizing, coding, and interpreting raw sensory 
1 Tm^ or experience-things seen, heard, felt, tasted, and smelled" (McCandless and Evans 1 973 p 
190). This group views the process called perception as having two steps: the first, the sensing or 
collecting of information; the second, the interpretation of that information or derivation of meaning 
Ip^'" 1 S^*^^""^^- Additionally, the two steps of the process are seen as being separate 

(Bigge 1964). On the other hand, a second group of psychologists would contend that perception is 
a unitary process and that sensing or collecting information occurs simultaneously with finding 
meaning or interpreting information. They consider perception to be highly selective since what 
information is collected depdnds on what it means to the individual and since that meaning is affected 
by the information that is collected (Bigge 1964). Kingsley (1957) partially bridges the gap by con- 
tending that perception ranges from the minimal levnl of merely gathering information or acknowledg- 
ing that something exists, to using past experiences and training to make appropriate interpretations. 
In any case, the major differences between the two groups seem to revolve around the issue of 
selectivity. 

A clear-cut definition of perceptions seems elusive since it is easier to determine what they are 
not rather than exactly what they are. However, in the strictest sense the term is being misused by 
those who claim to be coWectinq perceptions held by former students regarding the value of their 
training. What they are really interested in are the judgments or opinions that former students hold 
regarding the value of their training. While that distinction may be a rather technical point, it is 
quite important. The word perception connotes more validity and reliability than the words judgment 
and opinion imply. Therefore, it should be clearly noted that the information collected represents 
judgments or opinions-not perceptions, but the products of perceptions. 

Problems and Issues in Collecting and Using "Perceptions" 

One of the major problems in collecting and using "perceptions" has already been identified, 
that being that what are commonly called "perceptions" are not perceptions, but in fact judgments 
or opmions. As previously discussed, although it may seem to be semantic or technical trivia to make 
this distinction, it is not the case if using the term "perceptions" attributes more validity and reliability 
to the data than they deserve. When this major point is recognized, other problems or issues can be 
investigated. Much of what is discussed has its origin in the field of perceptual psychology and thus 
will deal vyith factors that affect actual perceptions and that are relevant to this discussion since 
judgments^ are based upon, though not synonymous with, perceptions. 



■ \ 

Individuali 



; One of the first problems to be faced is individuality; Every human being is a unique person, a 
fact often accepted in theory, but less often in practice.'lt affects all aspects of education and is ' 
particularly in^portarit when an attempt is being made to measure how an individual, feels about 
something. To Relieve that judaments can be reached with total objectivity is unrealistic./fcombs, 
Richards, and Richards (1976) state that individuals develop their unique "reality world/" on the 
basis of their id^^s, values, concepts, and beliefs about the world around them. The bases from which 
they make judgm^ents are a product of their background and experiences. Thus every individual uses 
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a different set of crit3ria when makinq evaluations and arriving at judgments. In some instances 
those differences are unimportant, in others very important. Differences are minimized if trained 
evaluators make the judgments. Kingsley (1975) points out that training can improve the validity 
and reliability of perceptions and therefore of judgments. That is, training can improve objectivity, 
and it is obvious that few former students have such training. - ' 



Values . 

V What an individual values has an effect on how events are perceived, and consequently on what 
judgments evolve concerning those events. Not only do the personal values of individuals affect 
their judgments or opinions, but the values of their peers or work group are also a factor. Dember 
and Warm (1979) conclude that experimentation indicates that the value attached to something 
affects how it is perceived, thus individuals who, b^ecause of their background, place a high value on 
education are likely to make more positive judgments about their training than individuals who value 
education less. Since individuals tend to reflect the values of their peer group, former students working 
m an environment where on-the-job training.and experiences are more respected than formal training 
tend to make less positive judgments than those working in environments where formal training is 
highly regarded by coworkers and is considered a key to promotion. 



Needs and Expectations 

Needs and expectations are closely related to, and have an effect on, the worth attributed to 
things and events. Probably the most important need any person has is the need to feel adequate, and 
to feel worthy. Evaluations that will protect one's feelings of adequacy are based upon wh^t the 
individual believes to exist. Thereforepersons who have invested considerable tfme and effort in 
completing a training program, especially if they have performed well in the program, have a tendency 
to justify that time and effort by rating the value of the training high. 

Fprdon (1976) indicates that any given characteristic of a training program may be a source of 
satisfaction or dissatisfaction depending upon the view individuals hold and their needs. Because, 
people enter jnto activities with certain expectationsV.egarding how the^activity will be conducted 
and its outcomes, these expectations tend to mold, or at least focus, their judgments as to the value 
of those activities. Bruner (1 973) points out that perceptions are a result of, and dependent upon, 
expectations, and that judgments are based on how well the expectations are fulfilled. It is impossible 
to know the expectations of all students who enter a vocational program or even whether their 
expectations are realistic^ All that is known is that they have expectations that will affect their 
judgment regarding the value of the program. 



Situational Factors " . ■ ' 

There are a number of situational factors that can and do have an impact on a person's judgments 
about an experience. The pre- and posttraining experiences combine to form the environment from 
which an individual makes judgments about value, and most of these experiences are outside the class- 
robm or lab and beyond the control of educators. Among the situational factors that may affect 
former^students' judgment regarding the value of a program are the following: their experiences in 
other educational programs prior to and following the program being evaluated; the economic 
climate and whether or not jobs are available; and the extent to which job requirements match train- 
ing: What individuals perceive is based on what they have learned to perceive, their past experience, 
and the current situation. ' . 



.*,k,* experiences prior to a vocational program are important factors in determining 

what kind of experience individuals expect to have during their training programs. Although those 
expectations are largely beyond the control of vocational teachers, they can have either a positive 
Of", "egat've effect on the students' evaluation of their training. Likewise, posttraining experiences 
with other educational proarams can affect former students' level of satisfaction with their training 
If they enter advanced training programs and do well, they are more likely to rate the program 
performance " '^^'"^^^"^ °^ ^^""^"'^ ^^^^ "^^V have contributed to their 

The number of jobs available and state of the economy are not only beyond the control of a 
vocational teacher but difficult to predict. Vocational programs, because of their purpose foster an 
expectation in many students that they will be able tc go to work in the trade area immediately after 
completing their training. Such expectations may or may not be realistic. However, if expectations 
are not fulfilled, the former students' satisfaction with the program will be affected. 

The matching of job to training is a particularly difficult problem for vocational programs that 
are generally designed to provide the basic skills and knowledge necessary fora typidal entry-level 
job m a particular trade or occupation. The job requirements of entry-level positions vary widely 
within the same occupation depending upon the industry or setting in which the individual is 
employed. This variability in the work place is beyond the control of educators but will undoubtedly 
affect how valuable former students consider their preparation to have been. . 

Quality of Training or Satisfaction with Training 

Another issue is whether former students' "perceptions" represent a measure of the quality of 
the training provided or merely a measure of their satisfaction with the training. In many instances 
the questions used to determine "perceptions" actually use either the word satisfaction or satisfied' 
Even though carefully worded questions may indicate that a judgment or opinion regarding quality is 
^at IS desired, it is a fairly safe assumption that much of what is measured is still satisfaction Hinds 
<1975, p. 43) cautions, "Beware of courses that classes enjoy." What is important is to find out why 
they enjoyed them. Was it because they learned useful knowledge and skills or because the instructor 
was an entertaining and nice person? 

Satisfaction is most often related to the attainment of personal goals and objectives that evolve 
from personal needs, values, and expectations. The attainment of those goals and objectives is often 
largely dependent upon opportunity and environmental or situational factors. The extent to which > 
students personal goals and objectives coincide with the goals and objectives of the training program 
IS generally unknown: However, it is safe to speculate that the match is most often considerablv less 
than perfect. . 

. Probably few vocational education programs have as a prirpary objective that students be satisfied 
with the training. As educators we would like to meet the stated program objectives and satisfy all 
students at the same time. However, that is not always possible and what is most important is to meet 
the program objectives, assuming that they are valid and reasonable. As Hinds (1975) points out the 
important aspect of evaluating training is not whether it was enjoyable but whether the students' 
learned something they could use. 

Thus it would seem that the degree to which former students are satisfied with the training 
provided is not necessarily a measure of the quality of that training. However, satisfaction data have 
been used and reported as if they were measures of quality and are sometimes more highly regarded 
than data measuring program quality. 



Measurement 

As a preface to the specific problems in measuring "perceptions." a review of the problems 
psyphologists have encountered in measuring actual perceptions is indicated. It is worthwhile to 
Identify those problems since what a person perceives forms the groundwork for the judgments or 
opinions often called "perceptions" or satisfaction. Combs. Richards, and Richards (1976) point out 
that the only way to learn how persons see themselves or the world around them is to ask them 
. sychologists call that technique introspection. Some of them have concluded that it is too inaccurate 
and unreliable to be useful. The following problems with introspection were identified: 

1. Degree of the subjects' awareness 

2. Errors in communication or misinterpretation — !>„^y 

3. Social expectancy (e.g.. peer pressure or the pressure to give socially 
acceptable answers) 

4. Cooperation of the subject (e.g.. openness, willingness to participate) 

5. Freedom from threat and degree of personal adequacy 

6. Change in field organization due to change in focus and thereby in the 
perception created when question regarding perception is po"5ed 

The specific problems program evaluators encounter when collecting former students' "perceptions" 
are in many ways Identical or related to those identified by psychologists. 

Although almost all follow-up questionnaires promise anonymity, the extent to which 
recipients of the questionnaires believe that anonymity actually exists is unclear. Probably, lack of 
faith m the condition of total anonymity is more universal than faith in its existence. Thus, to some 
degree, such questionnaires represent a threat to most recfpients. especially when opinions or 
judgments are sought, and this threat may afffSct individuals in various ways. \ 

First, they may choosfj not to respond. That raises the possibility of nonresponse bias in the 
data collected, since the judgments or opinions of those individuals who do not respond may be 
quite different from the opinions of those who do. The evaluator is then caught in a^dilemma. To use 
the data as collected rnay be to misrepresent the situation, but to contact nonrespondents in an effort 
to collect data may increase their feelings of being threatened or may reinforce the belief that anonymity 
does not really exist. However, it is generally agreed that it is better to attempt to.collect data frOm 
nonrespondents and to determine the extent of nonresponse bias, if it exists. 

The second way recipients may be affected, is to choose to respond by providing socially 
acceptable or what they believe to be the desired responses. Such a situation creates response bias. 
Rosenberg (1964) concludes that such a situation is a result of evaluation apprehension, an "active, 
anxiety-toned concern" on the part of the subject to win a positive evaluation of personal adequacy 
or at least to provide no ground for a negative evjaluation. It would seem that response bias generally 
results in responses that are rfiore positive than attually warranted. However, that is not always the .. 
case. > . ^ 

A final issue is the question of motive. There is some evidence to indicate that people with very 
strong feelings, either positive or negative, respond more readily than less opinionated individuals. 
Their motives for responding may be negative or positive. Additionally, respondents attribute some 
of their motives to the manner in which the evaluation was conducted. Thus, the motive of the 
respondents and their interpretation of tKe motives for the evaluation may also create a response 
bias. , . 
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ic h. J ? ^ satisfaction, in itself, is questionable. Lokiec (1973) states, "Human satisfaction 
IS brief and is never complete" (p. 988). Satisfaction is largely a product of time and circumstance 
and as these change, so too may level of satisfaction. Pre- and posttraining experiences affect level of 
satisfaction, and it is impossible to measure their effect accurately or to negate them Likewise^ 
satisfaction and judgments about value may be affected by time. They may well depend upon whether 
enough time has elapsed to allow the benefits, if any, of the training to occur. In the absence of 
sufficient time, former students must deal with abstract rather than immediate gratification The 
degree to which they are able to do so will affect their level of satisfaction and their evaluation of 
the worth of the program. 

There are three additional issues that affect the reliability of the data called satisfaction with 
training or "perceptions." First is the fact that these data are based on memory. Learning theory 
includes the law of intensity, which states that people remember best and longest those things that 
are unusual, dramatic, traumatic, or exciting, and forget those that are routine and normal Thus as 
time increases between the actual experience and its evaluation, the impact of unusual events ' 
positive or negative, will have an increased effect on the evaluation. ' . 

Second is the question of how accurately satisfaction or worth can be measured. Only relative 
values can be bsed to measure satisfaction or worth and those stem from the inclination and cultural 
background of the individual and the informal normals established by the peer group (Lokiec 1973) 
Additionally, the effectiveness of the device used to measure satisfaction or worth and the interpre- ' 
tation of results depend on the values of the person constructing the instrument and interpreting the 
results. There is considerable room for misinterpretation since the values of the evaluator may or 
may not match those of, or even be meaningful to, the respondent. 

Finally, there is some evidence to indicate that when satisfaction or "perceptions" of worth are 
measured they change. Combs, Richards, and Richards (1976) contend that they change because the 
focus IS on reaching a judgment. Judgments are multidimensional and the process used to arrive at 
a judgment involves making a series of complex tradeoffs (Lawson 1977). Being involved in such a 
process may change somewhat the very thing that was the original object of measurement. 

How "Perceptions" Should Be Measured and Used 

Before methods qf measurement are decided, it would be best to determine whether "perceptions' 
must be measured. Although there is no specific federal requirement to 'assess former stucients' 
"perceptions" of the value or quality of their training or of their satisfaction with their training, the - 
Education Amendments of 1976 require the evaluation of planning and operational processes including 
quality end availability of instructional processes^ Some vocational educators feel that former students' 
perceptions with their training provide valuable data in mefeting this requirement of the Act. However, 



collection of such data is a complex undertaking 
Measurement 



The first step in planning any evaluation is to define its purpose. This definition will provide 
guidance in determining what data need to be collected and how they should be collected. Only those 
data necessary to fulfill the purposes of the evaluation should be collected. Thus, if one of the purposes 
of the evaluation effort is to determine how former students feel about their training, satisfaction or 
"perceptions" data should be collected; but if that is not a purpose of the evaluation, they should not 
b.e collected. If it is decided that satisfaction with training or "perceptions" should be collected, the 
next problem is to identify the procedures most likely to produce reasonably valid and reliable ' 
information. . . 
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One of the first things that should be done when collealng Information Is to make clear to the' 
respondent exactly what is being asked. Thus it vyould be advantageous to call ' Weptions'' what 
they really are, rather than to confuse the issue With impressive terminology. What have commonlv 

trtinina ifth T'^^' ^"'^"^^"'^ °'' man? instances ri famTon with 

IZll Jr' ° 'V' '''"T '° ^han the word perception because 

individuals are more likely to understand the type of information desired and the process to be 
followed m determining their response if the information sought is correctly identified. 

rpii»hM!l?/^^w^^K"' "''l.ue-one which is usually, but not always, possible-for improving the 
reliability and validity of the information gathered is to specify the criteria tc be used In arriving at 
airrlTnTin^f ''""^'"^ evaluation. While it is still impossible to certify that 

all respondents used the same criteria to arrive at their conclusions they, at least, were instructed to 
^ta^ C^w ^'"i T^''^ establishes a mind set for the respondent,' helps to clarify exactly what is 
5 being asked, and e.iminates the need for guessing how the question should be answered Addiiionallv 
^ It may also reduce or eliminate sources of personal bias such as attitude, mood, or recent expSce 

Finally, there is the question as to when the former students' judgments, opinions, or level of 
^tisfaction should be measured. Enough time must elapse that the effects of the training program 
have had a reasonable chance to develop; however, too long an interval may cloud the respondents' 
memories. The time when data should be collected is best determined by reviewing the goals and 
objectives of the program and deciding when the effects of training may be expected to develop 



Use of Information 



Before a discussion of how former students' judgments .of tlie value of their training and satis- 
faction with It can be properly used, the validity; of such information must be briefly reviewed In 
evaluating the extent to which "perceptions of.effeatiyenessf^of training are related to actual evidence 
of effectiveness, Blumenfeld and Crane (1973) fbund that ntfiblationship exists. They therefore con- 
cluded that perceptions" are opinions, not evidence of effectt\/eness, and should be recognized as 
such. Since the validity and reliability of sdch data are so^mevvhat suspect, they should be used with 
extreme caution and for limited purposes. ' 

Blumenfeld and Crane (1973) suggest thatjthe utility of such information can be increased by 
supporting it with more specific data. Specifically, they recommend the use of a pretest posttest 
research design with a control group. A researchl design of that nature provides the kinds of data 
needed for accurate assessment of program effectiveness and serves as a basis for evaluating the reality 
of former students' judgments and opinions. This approach is reinforced by Parker (1973) although 
his recommendations are more general. To gauge program effectiveness adequately Parker 'suggests 
that four kinds of data are-needed: posttraining jbb performance data, posttraining group performance 
data (1.^,, how the training affected the group the Individual worked with), participant satisfaction 
data, ahd participant knowledge-gain data. BlumenWeld and Crane would contend that vvithput pretest 
data and a control group such an evaluation design ^ inadequate. However, it represents a definite 
improvement over the use of subjective data and opinions alone and is more likely to be used than ' 
the more scientific pretest, posttest research design With a control group. 



One key to piroper use of former students' evaluations of their training is to use the information 
for the specific purposes for which it was collected anl^ to rebort it in the form in which it was 
collected. The validity and reliability of such data are suspect enough that to use them in post hoc 
fishing expeditions, especially of a cause-effect nature,\is risky, and perhaps unethical Reporting the 
information in a different form than that in which it was collected also raises questions of ethics and 



propriety,. One common questio lable procedure is the combining of response categories or options 
If data are collected on a four-point scale of excellent, good, fair, and poor they should be reported 
on that four-point scale. To combine categories such as excellent with good and fair with poor rf^sks 
the actual results. Evaluators must be willing to report results in the most accurate, straightforwart 
manner. " ^ 

VVhen former students' evaluations of their training are reported, some explanation and inter- 
pretation should be provided. It is unfair and discourteous to the reader to do otherwise. Explana\ 
should include how the data were collected, including the actual questions asked, and should alert 
the reader to the possible sources and types of bias that the data may include. Moreover, a reasonable 
and honest interpretation of what the data actually do and do not mean should be provided including 
some discussion of what the data mean, for example, in light of such.factors as the state of the economy 
or currejnjob market. It is important that those w,ho may use the data understand their limitations.- 
iimitations that can- best be defined by whoever collected the data. 

The judgments, opinions, and level of satisfaction of former students regarding their training are 
primarily useful for two purposes. They serve to indicate areas or aspects of the program needing 
further study; and although they do not provide the final answers, they can provide early warning of • 
problems or indicate where possible strengths lie. Thus, they serve as a focus for more indepth investi- 
gations. Secondly, they have public relations potential. The data can be used to improve the image 
of the program within the ethical limitations of the data. The very fact that such data are collected 
can build good public relations, since many people like to be asked their opinions and like to feel that 
fducators are interested in obtaining their evaluations. For that reason alone, the complex process of 
data collection may be worthwhile. . 

Summary 

When the effectiveness of programs is evaluated, one of the types of data often collected is . 
former students' satisfaction with their training in terms of their "perceptions" of the value of their 
experiences in the program. In fact, the term "perceptions" is a misnomer. The data collected are 
actually former students' judgrhents and opinions regarding their training and, as such, are subject to 
; the influence of a variety of factors and circumstances, most of which are beyond the control of 
educators. Additionally, the extent to which such factors and circumstances afreet former students' 
judgments is unknown. Thus, the quality of such information is questionable and must be used with 
caution. 

There are a number of problems and issues regarding the collection and use of former students' 
judgments of the value of their training. Many of these problems and issues are related to factors 
affecting the validity and reliability of such data, for example, response and nonresponse bias, the 
situational Mature of judgments, and how responses are affected by an individual's values, needs, 
^ expectations, motives, goals, and objectives. How these data are used is also at \ssu% since evidence 
indicates that they neither provide final definitive answers nor indicate cause and effect relationships. 
Furthermore, high levels of satisfaction with training experiences or positive judgments regarding the 
^ value of training are not, and should not be, used as indi jators of program effectiveness or quality. 
^ ~-' ■ . . •- 

It is not mandatory that fprmer students be given the opportunity to evaluate their training. 
Whether those evaluations should be collected is not an easily answered question. Neither is the ' 
question of how they should be collected. Guidelines for collecting such information include the 
following: only those data directly related to the stated objectives of tho evaluation should be 
collected; the information should be collected as opinions or judgments and treated as such; efforts . 

' . " . > , 
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should be made to collect data that are specific rather than general and those that are related to 
program goals ?nd objectives; the criteria to be used in reaching a judgment should be specified 
whenever possible; and the time for data collection can best be based upon the goals arid dbjectives 
of the program and on common sense. , - J 

. The value of this type of data is limited, especially when it is used without supporting data of 

ti^rS/h " "'^''y ^yP^ °^ "^'^^ ^^'^ be collected in a comprehensive evalua- 

tion. When data of this nature are reported they should always reported in the form In which 
they were collected and be accompanied by explanations and interpretations. Potential users of the 
data should be cautioned^bout their limitations. Measures of former students' Judgments regarding 
the value of their training or satisfaction with their training are primarily valuable for two purposes 
to indicate needs for further research and to improve building public relations. In combination with 
other data they have the potential for.assisting in making decisions aL^Jut program improvement 
and for addressing accountability. , 
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MEASURING JOB SATISFACTION IN THE 
EVALUATION OF VOCATIONAL EDUCATION 



by 

Robert S, Billings* 



An evaluation of the impact of vocational education may include an assessment of the iob ' 

Sfv?na°t"h^ P^P^^ ^° f^'^'"*^^^ ^"'^h an assessment by |1) 

clarifying the nature of job satisfaction and its causes, (2) discussing the alternative questions that 

a^'nd iob tuf' r '"'S"' ''•^'^"^^'"^ the possible relationship bitween vocational eduS ' 
nJll c^h \ <4>.^"9^«.«'"9 and evaluating some research designs,-and (5) discussing a series 

of issues that may arise in designing the instrument such as selecting an existing nstruSi oXosed 
to designing a new one, or deciding what item format to use. Throughout, a basic theTe emerges 
the choices to be made depend largely on the purpose of/the evaluation. ^ ' 

Definition of Job Satisfaction 
The most common definition of the concept of job satisfaction is as follows: 

The affective orientation toward or emotional reaction to the job (or various 
components of the job) resulting from the appraisal or evaluation of one's job 
or job experience against the standard of one's values (Lawler 1973 and ' 
Locke 1976).' 

This definition emphasizes two interrelated points. First, job satisfaction is a feeling or an 
emotion, and as such, is likely to have a strong impact on the lives of employees. Although job ' 
tTJtirTi''^" emotion, it is not without foundation. The second part of the definition suggests 
that the feeling of satisfaction stems from a (more or less) careful evaluation of one's job-what it 
has to offer in various ways. Stated in a diff erent way, there is a descriptive part of one's attitude 
toward the job (What is the job like? Wh3t does it offer?) and also an evaluatTve coXnen"(How 
satisfied are you with what the job offers?). • ' 

Job satisfaction can refer to the overall satisfaction with the job or to satisfaction with specific 
components (or facets or elements) of the job, such as pay, promotion opportunities, supervision 
and so forth. T.,ese components can be fairly genera! {e.tj., the nature of the work itself) or much' 
more specif (c (e.g., degree of autonomy allowed in deciding what procedures to use when doina a 
specific task on the job). 



A list of the most commonly used oomponsRts of fie job" would probably include most of the 
following, although the list is illu.strativs an'i not exhaustive (summarized from Campbell and 
Pntchard 1976; Lawler 1973; Locke 1976). It includes work itself, (intrinsic interest, variety oppor- 
tunity.for learning, difficulty, amount to be accomplished, chances for success, control cr autonomy 
of pace and methods, responsibility, use of abilities, feeling of achievement, opportunity to stay 
busy, authority, chance to do things for others, status or prestige); pay.(amount, fairness or equity 
method of payment, possibiliiy of future raises); promotion (opportunities, basis for determining 
advancement, fairness, type offered); recognition (praise from supervisor, coworkers outsiders)- 
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V). t^^j 



>Arork.ng conditions (hours, rest oreaks, equipment, temperature, ventilation, humidity, location ^ 
physical layout); supervision (overall styie, influence over others in work place, technical skills 
human-relations skills, administrative skills, consistency); coworkers (competence, friendliness ■ 
helpfulness); and company policies (concern for employees, specific policies toward unions retire- 
ment, transfers, reputation of company, job security). ' 



A Model of the Causes of Job Satisfaction 
Satisfaction With Job Components 

Lawler (1 973) has summarized and evaluated the various models of the causes of job satisfaction 
concluding with a consolidated model that has gained wide acceptance. A somewhat simplified 
version of this model is presented in figure 1 and discUssed in some detail below in the following 
paragraphs. ^ 

Earlier it was Suggested that the amoum of job satisfaction results from an evaluation of the 
job. A pursuit of this point results in general agreement that level of satisfaction (overall or with a 
specific component) is a function of the discrepancy between what the person feels should be and 
what IS sfeen as existing. With pay as an example, if an employee feels that compensation is less than 
tie work merits, then that employee will be dissatisfied with pay. If the "should be" and "is" are in 
balance, the employee will be satisfied. If the amount received is greater than the employee believes 
the work deserves, a feeling of "guilt" or "over-compensation" results. (The discrepancy in this 
direction probably has to be fairly large for a significant feeling of guilt to occur, particulary for an 
aspect of the work like pay.) This part of a model of job satisfaction is substantiated by empirical 
evidence. . 

Although this model is fairly straightforward, it has some interesting implications. For example 
two employees may have, objectively, the same opportunity for promotion and yet differ in satis- 
faction with this promotion opportunity. The differences in satisfaction may be due to (1) different 
perceptions of the opportunity for promotion (the "perceived amount" part of the model) and (2) 
different standards concerning what the chances for promotion "should be." This example emphasizes 
the pomt that the "should be" and "is" part of the model are judgments and are not necessarily 
objective or accurate. ; 

This model of job satisfaction also includes the determinants of the "should be" and "is" 
received perceptions. One of the determinants of "should be" are perceived personal job inputs ' 
These include all of those things that individuals feel they contribute to the job such as skill 
experience, effort, performance, education, and so forth (see figure 1). The higher these perceived 
inputs, the greater the amount of wprk reward individuals feel they should receive. For example if 
employees feel that they perform better than coworkers, then perceived inputs are high and the ' 
amount of outcomes that should bejreceived is correspondingly affected. 

The model suggests that perceived job characteristics also affect the perceived amount that 
should be received. The more demanding the job (e.g., the higher the level, the more difficult the 
work, the more responsibility required), the greater the outcomes (e.g., pay, intrinsic satisfaction 
working conditions) that should be received from the job. For example, an employee who is given 
new, demanding job duties is likely to feel that the pay should be greater. 

The determinants of perceived amount of the outcome that is received are simpler (see figure 1 ). 
Actual outcomes, of course, have the greatest impact on this perception. In addition, however, the 
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FIGURE 1 

Lawler's Model of the.Determinants 
of Satisfaction with Outcomes 
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E. E. Lawler, Mmtion in ^ork Organizations (Belmont, CA: Wadsworth, 1973).Permission,to reprint being requested. 



the amount of the outcome believed to be received by otiiers is important. Take as an example twc 
computer programmers, each earning $20,000 a year. If programmer A's coworkers earn $18 000 
then A will feel that the perceived amount of pay is relatively high. Programmer B however works 
alongside other programmers wl^o make $22,000. B will perceive the current level of pay to be 
lower. Thus, assuming all other factors in the model to be equal for A and B, programmer A will be 
more satisfied with pay than B. 

Before other models of satisfaction are examined and applied to vocational education, it should 
be noted that this model has received a fair amount of empirical support. The findings of many 
older studies can be reinterpreted in this framework and are generally consistent with the theory 
In addition, a recent study by Dyer and Theriault (1976) specifically tested most of their model and 
found general support. 



Overall Job Satisfaction 

It is often hypothesized that overall job satisfaction is a result of some combination of satis- 
faction with the various components of the job. Further, it seems reasonable that the components 
most important to the individual somehow are weighted more than unimportant components in 
determining overall satisfaction. For example, an employee who values challenging work more than 
friendly coworkers will be more satisfied overall if satisfied with the nature of the work and dissatis- 
fied with coworkers than if the opposite were the case. 

Many studies have been conducted to test this suggested relationship of component satisfaction, 
importanceof the components, and overall job satisfaction (Quinn and Mangione 1973; see Locke 
1976 for a complete summary). The empirical results suggfest a more complicated relationship. 
Evidently, individuals' take importance into account when rating component satisfaction; important 
components are usually rated either very satisfying or very dissatisfying, while unimportant compo- 
nents are rated as more neutral. Thus, a separate rating of component importance is redundant with 
the component satisfaction rating. This redundancy implies that a simple sum of component satis- 
faction ratings is the best predictor of overall job satisfaction. However, it is wise to note that since 
important components are rated in a more extreme manner, the variability in the sum of the compo- 
nent satisfaction is mostly a product of the ratings of the important components. Thus, the importance 
of a component is valuable information and does have an impact on overai! satisfaction„albeit indirect. 

Regardless of the precise manner in which component satisfaction'is combined to result in 
overall satisfaction, this way of viewing overall job satisfaction suggests that Lawler's model of 
component satisfaction also explains the causes of overall satisfaction That is, the employee (1) 
judges, for ?ach component of the job, the amount that should be received, (2) judges the amount 
that is received, (3) feels satisfied or dissatisfied as a result, and (4) combines all of the evaluations 
across components to determine overall satisfaction. 



• Issues To Be Considered 

Having defined job satisfaction and presented a model of its causes, we now turn to a list of 
questions that should be addressed when evaluating the effects of vocational education on job 
satisfaction. ' 
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What is the Purpose of the Evaluation? 

The first step in determining the purpose is to clarify the major questions that auide the 
TnTr- T'^t °^ job satisfaction selected for inclusion aid the sSc d^ gn ui S 
depend upon the precise question being asked, . ^ "'^■a" "sea 

Two major purposes seem likely: 

1. What are the Effects of vocational education as opposed to traditional education? 

2. What are the effects of certain characteristics of the vocational education program? 
The first question would probably be the major focus if the purpose is to justify the existence of ' 
the vocational education program, justify an increase in the size of the programyavoiS in the 

fhp'nnr'^n' Z '''f ""^^ 'H '''"'^ ^ The second question ^^^d apply if 

the purpose was to gather data to make decisions about the operation of the prdgrarExaSs 
of such decisions would include the amount of field experience given to students th^ type of 
instructor hired, the training programs offered, and the content of curriculum 

While an evaluation could attempt to answer both questions simultaneously, the two questions 
fesja'Ici'destgns ''''''' °' ^'"^ ''''T''' '° TfSenT 

What Dimensions of Job Satisfaction are 
Expected to be Affected and Why? 

The answer to the question of job-satisfaction dimensions depends upon the purpose of the 
evaluation. Accordingly, we will examine the two major purposes separately. : 

Effecte of vocational education as opposed to traditional education. If the major purpose of 
nupftinn r " 'l^'f^"^'' °f vocational versus traditional education, then the following 
ZtflZ 'n f H « ^«^;!°"3'^ducation students do not receive that education, how will their job^ 
satisfaction be different? There are several possible effects: 

/. Vocaiional education may decrease job satisfaction by increasing perceived inputs but not 
affecting perceived outcomes. 

This possibility follows Lawler's model of component satisfaction. In general, we can predict that 
nprhlnc*^'h° « ^^1" increase job inputs, such as skills, training, education, experience, and 

perhaps job effort and performance. If job inputs are perceived as higher, then the voca^onal education 
^graduates will feel that they should be receiving more and better work outcomes, ,uch as better pa^ 
more promotion opportunities, more intrinsically interesting jobiassignments, more autonomy, and 
better working conditions. In other words, vocational education may increase the level of aspiration 
for most to acquire more, if not all, work outcomes. The paradoxical conclusion is that better training 
.may lead to lower satisfaction-if the heightened aspirations of "should be's"are not matched by the 
actual level of outcomes. 

A further implication is that this "aspiration level" effect should be explored when evaluating 
vocational education. One simple way to do so is to ask graduates as to whether they feel that 
vocational education should mean more and better work.outcomes. If so, exactly what do they 
think should be better-pay, promotion oppprtunities, autonomy? Finally were their work outcomes 
Detter-do they feel that they got better pay, because they were vocational education graduates? An 



analysis of these types of perceptions would help explain a lack of effect or^ negative effect of 
vocational education on job satisfaction and also suggest which dimensions should be examined ' 
Further, this may have more serious implications when assessing the job satisfaction of special 
populations such as the handicapped and disadvantaged individuals. 

2. Vocat ional education may increase job satisfaction because it leads to better jobs 
for its graduates. 

If this is the presumed mechanism behind the effects of vocational education on job satisfaction 
then the logical thing to do is to demonstrate that the jobs obtained are different Further the ' 
di mensions of Job satisfaction to be examined are therefore those that deal with the job charac- 
teristics that are better because of better placement. 

The following elements of the job might be expected to be better due to vocational education 
depending upon the circumstances: pay, promotion, working conditions, and the work itself Other 
job characteristics would seem less likely to be affected: the nature and quality of supervision 
relations with coworkers, and company policy .which would probably be more strongly affected by 
the specific organization than' by the level or quality of the job obtained. 

It must be noted that the dimensions of job satisfaction that are affected by vocational education 
will depend heavily on the types of jobs involved. Further, if the objective of vocational education 
IS better placement, then the key issues are the precise differences between the jobs obtained by 
vocational education graduates versus those of traditional program graduates. The decision as to 
which dimensions of job satisfaction to include in the evaluation will depend heavily on a prior 
analysis of this issue. " 

3. Vocational education may have no effect on fob satisfaction. 

There are several reasons why job satisfaction-may not be affected' by vocational education. First 
overall job satisfaction, being a composite of more specific satisfactions,/ may not be affected if ' 
satisfaction with some dimensions is enhanced (due to better placement), while satisfaction with 
other dimensions is reduced (due to unfulfilled expectations). Further, vocational education may 
affect satisfaction with !^ome aspects of the job, while not affecting others. For example, satisfaction 
with supervision may not be affected. Such a pattern would dilute the.impact on overall' satisfaction. 
In addition, vocational education may increase the satisfaction with some components, have no 
efUct on others, and decrease satisfaction with yet oth^r components, due to the incre'ased-aspiration 
mechanism previously discussed. To the extent that these different effects occur, the total effect on 
overall job satisfaction becomes unpredictable. 

Job satisfaction may be unaffected by vocational education because job placement depends 
upon the availability of better jobs, and upon the assumption that better jobs are more likely to be 
obtained following vocational education. If vocational education graduates do have an advantage 
in competing for better jobs, then this happens because employers believe that vocational education 
graduates make better employees. Both of these variables-the rate of placement of vocational versus 
traditional graduates in good jobs and the beliefs pf employers concerning vocational education , 
graduates-can and should be assessed if a "job-placement" mechanism is believed to be occurring. 

The foregoing discussion of these processes is summarized in figure 2. Note that the two opposing 
mechanisms arp shown: (1 ) vocational education may raise job satisfaction by providing better job 
outcomes through better placement, or (2) it may lower job satisfaction by raismg expectations' that 
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FIGURE 2 

The Possible Effects of Vocational versus 
Traditional Education on Job .Satisfaction 
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are not mst. The key variables that determine which mechanism predominates seem to be the 
availability of better jobs and the likelihood that vocational education graduates actually will 
receive these jobs. ' ^ 

Effects of characteristics of the vocational ediication program. We now turn to the second 
broad question guiding the evaluation-what are the effects of certain characteristics of the 
vocational education program? First, why should such characteristics affect the job satisfaction 
of graduates? One possible assertion to begin with is that the job placement mechanism seems even 
less likely to be operating here than in explaining the effects of vocational versus traditional 
education. In order for such characteristics as extent of fieldwork and background of the teacher 
■to influence, an employer's decision to hire a vocational education graduate, the employer must be 
fH^-? characteristics of the program and believe that they will produce better graduates. 

While this sort of effect is possible. It seems unlikely. Also, if a given characteristic is altered then 
It takes some time for the change to affect the job satisfaction of graduates through the job place- 
ment mechanism. - / ■ 



A characteristic that .may produce an effect on job satisfaction is the opportunity provided by 
vocational education for a realistic preview of the job or occupation!, thus allowing students to 
decide whether that occupation fits their needs before taking the job. For example, nursing aide 
students may discover that the job does not hold as much interest as they thought after finding out 
during training,.what the job duties will be. Ary nteresting implication of this point is that in some' 
cases. It may be a positive outcome if individuals learn during training, rather than after job placement 
that they are not going to like the job. . f . 



The realistic job-preview literature, which is well summarized by Wanous (1980) would suggest 
that job satisfaction (and lower turnover rates) are produced by providing complete and accurate 
information-both positive and negative-about the prospective job. Applied to vocational education 
the implication is that certain characteristics of the program may enhance subsequent job satisfaction 
Specifically, any element of a program that helps provide an accurate picture of the vocation will 
make job satisfaction more like^. Such elements include instructors who have axper'ience across a 
variety of organizations and positions in the vocation, opportunities to spend lime in an organization 
doing the actual work (provided the position and organization are representative and the work done 
IS an accurate sample), and curriculum content that explores both the positive and negative charac- 
teristics of the vocation. 

We can now turn to which dimensions of job satisfaction are likely to be affected by a vocational 
education program providing a realistic job preview. Ressarch iri this area, as summarized by Wanous 
(1980), found that overall job satisfaction has often been enhanced by fairly short and simple realistic 
preview interventions (e.g., booklets or films about the job, which give a balanced view). It is, there- 
fore, not unreasonable to expect that a good vocational education program, which is much longer 
and more detailed, will have an even greater effect oh realistic expectations and job satisfaction. This 
effect, however, likewise assumes that those who experience poorly managed vocational education 
programs epter the vocation with either uncertain or overlV optimistic expectations. 

. It is more difficult to speculate as to which specific, as opposed to overall, dimensions of job 
satisfaction are likely to be affected by vocational education practices providing a realistic preview. 
It would seem to depend upon which aspect of the vocation would be perceived unrealistically due 
to poor vocational eduration. I could innagine a vocational education program that emphasizes only 
the interesting and challenging aspects of the work, because these are fun to teach and make it easier • 
to maintain student interest. However, such an ajaproach is likely to reduce the satisfaction with the 
work itself, ff the subsequent jobs are disappointing in challenge and variety. Also, it is possible to 
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imagine that a poor vocational educatic. . " ^qram may create unrealistic expectation? about 
promotion and career opportunities, pay ieve.,, add working conditions. 

It is up to those designing the evaluation to analyze fully the intended effects of the program 
elements to be evaluated. Perhaps a survey of -vocational education graduates will identify areas 
producing unfulfilled expectations, will iuggest certain program changes, and will guide the subse- 
quent" choice of job satisfaction measures for the evaluation, 

A Final Comment 

Writing a paper such as this Is somewhat frustrating, for there are too many issues that are 
important and too little space. Conducting evaluation research is a most difficult task because the 
entire process of defining the problem, operafionalizing the variables, determining the research 
design, putting together the instrument, collecting the data, and making sense of the results requires 
multiple skills. Moreover, therfi^e a frustrating number of choices to be made with inadequate 
information. If this paper has provided some insight into any of these difficult choices then my 
fjoal has been accomplished. 
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CHAPTER III 
SOME BASIC EVALUATION CONSIDERATIONS 

Michael, Patton (1 978, p. 22) bewails ths fact that the "emergence of evaluation research has 
not meant a correspondmg utilization of findings for rational decision making." A recent review of 
ilerature shows that there is great dissatisfaction with the lack of impact and usefulness of evaluation 
mfora)ation{Alkm 1980,p. 14). Carol Weiss (1972, pp. 10-11) savs: u ev-iuduun 

Evaluation research is meant for immediate and direct use in improving the 
/ quality of social programming. Yet a review of evaluation experience 

suggests that evaluation resu Its have not exerted significant influence on 
\ program decisions. 

• ... P- identified three majpr categories of factors influencing the ultimate 

utilizatron of evaluation findings: characteristics of the organization, characteristics of actors in the 
system (evaluators and decision makers), and characteristics oi the evaluation. A number of specific 
factors in the process account fdr ineffective utilization of evaluation data (Patton 1 978) • fuzcy 
program goals, lack of methodological rigor, uncertain findings, lack of staff, little program coopera- 
tion, inconsistent state and county data processing systems, unclear decision-making hierarchies 
political undercurrents, trying to cover too much, and inappropriate timing. Thus it can be seen' 
that, while utilization of evaluation information is influenced by several- variables, the evaluator's 
technical skill, ingenuity, and creativity constitute important determinants in the success of the 
utilization process. To a considerable degree, the evaluator determines what happens to the results » 
of evaluation. ' ■ / 



This chapter presents some basic evaluatiorl considerations to promote the utilization of data 
TOllected by measuring former vocational students' satisfaction with their training and their jobs 
This discussion includes (1) identifying relevant decision makers and information users (2) writing 
the objectives of the study and determining the respondents, (3) choosing the appropriate research 
design, and (4) deciding whether to design your own instrument or to select an existing one. 

Identifying tha Relevant Decision Makers 
' and information Users 

^ Unlike research studies that are designed primarily to add to the body of knowledge, evaluation 
studies are designed prinrtarily to provide good information for decision making (Alkin 1980 p 3) 
Other authors such as Weiss (1972), Patton (1978), and Stufflebeam (1971) echo the same view The 
SIX major purposes of evaluation identified by Anderson et al. (1975, pp. 3-4) reflect this "dominant 
viewpoint": 

1 . To contribute to decisions about program installation 
• 2. To contribute to decisions about program modification 
3. To obtain evidence to rally support for a program 



4. To obtain evidence to rally opposition to a program 

5. To contribute to the understanding of basic psychological, social, and other processes 

6. To contribute to decisions about program.continuation, expansion, or certification 

Thus, to increase the use of evaluation information, the joint Committee on Standards for 
Education Evaluation 1981, p. 13)'requires "evaluators to acquaint ithemselves with their audience, 
ascertain the audience s information needs, gear studies to these needs, and report the information 
clearly when it is needed." Three important questions, therefore, evolve: Who are relevant decision 
rnakers and information users? Whattvpe of information do different audiences need? When do 
they need the inrormation? Needless to say, you need to answer satisfactorily these questions before 
you proceed to do the other parts of your study; they are basic to the formulation of your objectives 
More importantly, your answers are cruciaj to the utilizaticnjof the information generated by your 

There are no simple, universal rules for answering the questions posed to you. As you may 
realize, different types of organizations have different types of decision-making hierarchies different 
political undercurrents, and different individuals responsible for decision making. In short different 
organizations have different environments. Thus, it might be helpful to view information needs in 
terms of accountability, decision making, and program improvement at different organizational levels 
-school, district, state, and federal. Toward this end, you may like to consult the following sources 
(Morris and Fitz-Gibbon 1978, pp. 23-24): ,) 
•i State and federal mandates and legislation 

• Local and national concerns identified in editorials, articles, and legislation 

• Parental concerns as voiced in letters, PTA and parent advisory meetings, and conferences 

• Community concerns voiced by busines':- organizations, pres-sure groups, and the like 

• School records and reports, such as attendance„discipline, career choices of qraduates 
and test scores - ' ; ; a ., . 

1-1 

• College requirements and employers' requirements for graduates 

• Teachers' reports and comments 

• [Former] student requests and comments 

• Project proposals, final evaluation reports, anci program descriptions of other projects 
in the same curriculum area ' . . I j - 

• Other schools that have developed programs.ln your^rea of interest 
Checklist 2 is designed to help you identify the relevant decisipn makers in'your organization. 

Writing the Objectives of Your Study 

... . I ,. ; ■ 

In the previous section, you were asked to answer questions pertaining to (1 ) the identification 
of decision makers in and information users of your oVganization, (2) the listing of specific informa- 
tion needs of your target audiences, and (3) the deter'minatioh of when your audiences need the 
information. Now, you are probably ready to proceeid to the/next phase of your study-writing the 
objectives. / 1 

Objectives are one of the most important parts/of youi] evaluation study. They'specify the 
desired outcomes; thus, they provide direction in terms of the data to collect, the time to collect 
them, the method or methods to employ, and the analysis ^nd presentation of datajcollected. 
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CHECKLIST 2 



Determining Appropriate Decision Makers 
and information Users 



Instructions: Examine the figures below and list the appropriate decision makers and information 
users for each phase of the vocational education program. 



Phase 1 



Phase 2 




Needs 
Assessment 



Formative 
Evaluation 



Appropriate 
Decision Makers 
and/ -\ 
(hformation; 
, Users 



1. 
2. 
3. 

1. 
2. 
3. 



Phase 3 Recycling 



Has the Program 
Had An 
Impact? 



Summative 
Evaluation 



1. 
2. 

3. 




Evaluation 
Audit 



1, 
2. 
3. 



Adapted from K. Adams and J. Walker. Improving tht Accountability of Career Education Programs: 
Evaluation Guidelines and Checklists. (Columbus, Ohio: The National Center for Research in 
Vocational Education, 1980). 1 
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In writing your objectives, you should keep in mind your target audiences and you should key 
your objectives to their data needs. To ensure that your objectives adequately reflect the intended 
outcomes, it is suggested that you draw a list of specific information needs of your target audience 
with similar items grouped together in a category; then, you translate your lists into objectives. ' 

..Your objectives may be classified into two categories: broad and specific objectives. Broad 
objectives express the major purpose of your study. Specific objectives, on the other Hand express 
the specific outcomes of your study. Several specific objectives may constitute one broad objective 
If specific objectives are written Jn measurable, concise, clear, and short statements, they can be 
the basis for stating questions to be included ,in the data collection j^njtruments. 

The example on table 1 illustrates how a list of specific data needs of different audiences is 
translated into broad and specific objectives. 

Now, make your list of specific information needs and of your target clientele and translate 
them into broad and specific objectives. After you have written your objectives, take time to review 
them by completing checklist 3. 



Determining Your Respondents 

After the objectives of your study have been vvritten, the next tasks are (1 ) to determinawhat 
relevant population(s) can furnish data to meet the objectives of your study, and (2) to identify the 
specific respondents from whom the data will be gathered. 

It is suggested that you study your objectives and make a |[st of different groups of former 
vocational students. Indicate in each group the number of students and the data needed. From this 
list, you should be able to determine the relevant populations (i.e.,jroups of former vocational 
students) of your study. An example is given below: 



Former Vocational Student 

Home Economics Class, 
1979-1980 

1. Total 



Number 



150 



Employed by the new food 
industries in the community 



50 



Data Needed 



• Satisfaction with the 
team-teaching approach . 

• Satisfaction with the 
new text books 

• Job satisfaction in terms 

of their pay, fringe benefits, 
and working conditions 



From the example, it can be seen that two types of information are needed from all former 
home economics students. However, for those who are employed by the new food industries in 
the community, an additional piece of information is going to be collected— job satisfaction. 

After you have determined your relevant population(s), your next job is to identify the specific 
respondents of your study, i.e., the selection of the former vocational students from whom data 
will be collected. Your decision depends upon several factors: objectives of your study, character- 



TABLE 1 

Translating Data Needs to Objectives: An Example 



Data iSleeds 

Student Satisfaction : 

1. Local appliance manufacturer-rwants to know the satisfaction of home economics students on 
the new appliances that were donated. 

2. Local school board--expresses interest in knowing the satisfaction of students on the n^w 
prototype instructtonal materials in business and office' education and health.occupatiohs 

3. Local union officials-want to know the training satisfaction of students who were in thi 
I apprenticeship program. 

Job Satisfaction: 

1. Parents-express interest in knowing the job satisfaction of former students who are employed 
V in the new factories in the community. 

2. Management of new factories in the communfty-vyants to know the satisfaction of their 
employees who were former vocational students relative to the various aspects of their jobs. 

\ ' \ • ■■ °- : ■■; • ■ 

Objectives ' 

Broad Objectives: 

To assist target audiences in decision making and planning by providing them with information on 
the following: : . 

A. Former vocational students' satisfaction with their train 
Specific Objectives: \ 

1. To measure the satisfaction of former vocational students on the new appliances donated 
by XYL Gpnipany. . . 

2. To measure the satisfaction of former vocational students in business and office education 
and health occupations oh the prototype instructional materials: 

3. To determine the satis^faction of former vocational students on the apprenticeship program. 

B. ' Fornier vocational students" satisfaction with their fobs. ^ ■ 

Specific Objectives: > . 

1. To determine the job satisfaction (relative to pay, fringe benefits-^ working conditions, 
status) of former vocational students in the newly-established factories in the community. 

2. To compare the level of satisfaction among former vocational students who are employed 
in the newly-established factories in the community. 
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CHECKLIST 3 
Objectives of the Study 



Instructions: Take time to revie.w your objectives by answering the questions below: 
Yes No Questions: 

1. Do the objectives reflect the needs of the following information users 
and decision makers? 



□ ■ 


□ 


• 


Parents 


□ 


□ 


9 


i' 

Teachers 


□ 


□ 


• 


Local school administrators 


□ 


□ 


• 


Business and industry 


□ 


□ 




Labor 


□ 


□ 




State education agency 



□ □ 



2. Do your broad objectives express the major purposes of your study 

3. Are your specific objectives — 

□ • Clearly written (i.e., they are free of words that are arhbiguous, 

indefinite, and imprecise)? 



□ □ 

□ □ 



• Concisely written (i.e., short and direct sentences)? 

• Measurable (i.e., th6 objectives are stated in performance terms)? 



istics of your population, and resources availabJe. In cases where your population is small (4rhaDS 
only wenty-five), you may decide to use a complete nnumeration survey On the ot^er kand vour 
popu ation may be large (e.g. 1000); thus, it becomes very expensive to gather Ja atS a , Imb^s 

thp Jot^!'^ T."- *° ""^'^ yoursel to a few respondents selected at'random f?om 

the total population of your study. Then, you use th^ information collected as if it^Dresent. thP 
answers of the whole population. Three' types of random samples are S used (^^^^^^ 

• fJZX'^"'^°T^T" *° "'^ P°P""ati-^n .Member a unique numbei^ select sample 
itemsby use of randpm numbers. i * / =>°'"mic 

• SK5femaf/c-Use natural ordering or order fiopulation; select rant^om starting point 
nf n«rL * "rarest integer to the sampling ratio (N/n);,select timesat interval 
of nearest integer to sampling ratio. / / 

• ninnZ'^fT.^""^^'"^ "^"^^ ^"'"^ °f ''^"^""^ sariipiing; ultimate units are 
groups; select these at random and t/ke a complete count of each. 

• 5fraf/y/ec/c/6?sfer-Select clusters aj'random from every sampling unit. 

• Repetitive multiple or sequential/-Jwo or more samples of any of the above types ) 
7rl necesary ' ' ^""^^^^ *° 'at^r ones, or determine if they | 

cnp.!fll^ ^een reached to select a random sample, the evaluator needs to determine the ' 

speci ic sampling strategy. To reach such a decision, the evaluator needs to^determine the type of i 
sample that best serves the objectives of the.study. For further discussion on sampling as it relates i 
rnpS°p"^ education follow-up studies, the reader may refer to Evaluation Handbook #1 : Guide- ^ 
lines and Practices for Follow-up Studies of Former Vocational Students (Franchak and Spirer 1978)! 

Checklist 4 is designed to help you determine your sampling plan. 

Choosing the Appropriate Research Design 

Earlier in this chapter, you were asked to study your objectivss and identify the procedure for 
selecting your respondents. Your next job may involve choosing an appropriate research design. In 
doing this, the evaluator needs to consider the objectives of the study, the evaluation questions that 
need answering, the characteristics of the population, the time available to conduct the study and 
the human and financial resources available. Additionally, you should consider the various rules and 
regulations thaj protect individuals' rights; for they set the "ethical" parameters for the methodologies 
that are available to the evaluator. ^ uai« 

For example, a local school board might be interested in knowing the effects of vocational 
versus nonvocational education in terms of the graduates' job satisfaction. The perfect design would 
involve conducting a true experiment, Students would be randomly assigned to vocational versus 
nonvocational education and subsequently placed in identical jobs in the same organization. Hovi/ever 
It IS unlikely that the researcher will have the freedom to make these assignments. Instead, as an alterna- 
tive design, an ex post facto research design is utilized. The job satisfaption of a group of vocational 
education graduates is compared to that of their coworkers in similar jobs in the same organization 
This procedure would entail selecting (or having the employer select) one nonvocational graduate for 
each vocational graduate in the sample. Although such an approach would begin to control for the 
effects of job and organizational characteristics, individual characteristics (values, age, experience and 
so forth) would not be controlled for, due to nonrandom assignment to groups. Even'with these ' 
problems, this is the best of the feasible research designs. 
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- CHECKLIST 4 
Determining Your San^pling Plan 



Groups 



Total N 



Decision 



A. Types of Program* , 

1. Agriculture " 

2. Business and Office 

3. Health Occupations 

4. Consumer and 
Homemaking 

5. Distributive 
Education 

6. Technrcal 

7. Trade and Industrial ' 

B. Special Needs Population 
' 1. Handicapped 

2. Disadvantaged 

3. Minority 

.4. Limited English 
Proficiency 




Total - 
Eriumera'tion ' 


Simple 
Random 


cluster 


Strati-, 
fied 




(Check One) 


•> 


f 1 

□ 


□ ■ 






□ 




□ 


□ 


□ 


n 


□ 


□ 


□ 




□ 


n 


1 1 

■LJ 


□ ■ 


□ 


n 


□ 


□ 


□ 


□ 


n 




□ 


a ,.. 










n. 


□ ■ . 


v.. 


n'- 


□ 


□ 


□ 


□. 




. □ 


□ 


□ 




□ 


□ 


□ ■ 



Depending on the objectives of your study, you may want to group your respondents by 
U.S. Department of Education 6-digit code. 
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Let us take another example. Your local superintendent wants to assess the effects of charac- 
teristics of the vocational education program. When the purpose is to evaluate a specific aspect of 
the vocational education program, then the ideal design would be to run parallel programs differing 
only in that one aspect, and to randomly assign students to the two programs. Whereas this design 
may be difficult to implement, it is not impossible, particularly where there are multiple classes and 
where assignment to classes is controlled by the administrator. If the effects of a program element 
are truly of interest, then a true experiment should be.carefully considered. The strengths of this 
design cannot be overemphasized because any differences in subsequent job satisfaction can be 
inferred to have been caused by the differences between the programs.-^ 
• ';/'•■ > > - • 

An alternative design involves nonrandom assignment to the experimental and control qroups 
producing a nonequivalent control-group design. For example, when two classes differ on the ' 
characteristic under study, but have been placed by nonrandom assignment, a number of differences ' 
exist besides the one of interest. Another possibility is to study two groups of students, one that 
went through the program before a change and another, after the change. 

_ Both of these examples produce groups that differ on variables other than the one under study 
The characteristics of the students or the availability of good jobs may have changed over time or 
the students may have self-selected themselves into the two different classes, on the basis of extrane- 
ous factors or on their knowledge of the design element being evaluated. Biases such as these may 
mask or overpower true effects or produce differences that are pot due to the variable being studied. 

A design that is generally stronger than the nonequivalent control grOup is an interrupted : • »- 
series design. If a change in the program is planned, then this design requires taking periodic w es 
before and after the change. Any significant differences in the pattern of the data from before : 
after the change can be used as evidence of the effects of the change. With an adequate timelir..^, the 
only threat to the conclusion. based on this design would be other events occurring about the time of 
the change that might have caused the effects. It is posssibie to reduce the plausibility of this alternat'iv 
by closely monitoring the program to make sure no other changes occur when the focal change is 
Implemented. , 

The interrupted time-series design is most useful where measures can be easily taken on a regular 
basis, eor example, attendance in class, scores on weekly standardized quizzes, and student reaction 
to the course are quite possible to use in this design. Job satisfaction, however, would seem to be 
more diff icult to measure with this design, primarily because before and after measures of job satis- 
faction cannot be taken from the same set of students. Rather, each student either graduates from a 
course taught before the change or a cdurse taught after the change. This destroys the advantage of 
interrupted time-series design, for subjects can no longer act as their own control. Accordingly, if a 
true experiment is not possible, a nonequivalent control group design is thebest d sign a researcher 
can use when examining the effects of vocational. education on job satisfaction. 

. The foregoing examples illustrate the fact that in all instances, the evaluator's idealism needs to 
be tempered by the realities of the situation. 

^ ■ ' ■ " - 

Another important consideration in choosing a methodology is data timeliness. This means that 
your information should be available at the time it is needed by your target audience. "Of what use 
is the grass if the horse is dead?" is an old saying. Currently applied, it becomes. "Of what use are 
evalu^n data, no matter how good, if they are not available at the time they are needed by decision 
makers?" In some instances, therefore, you may have to select a "less desirable".meth6dology using 
only "soft" data because of certain constraints, such as available time and resources. Patton (1978 
p. 237) says:, ,, , - ' 
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The' challenge in selecting an evaluation method Is to do justice to the question 
by providing the most valid and reliable answers possible with the resources 
a/a/Za/^/e [italics added] 



While some evajLators may find it difficult to sacrifice their "research integrity" for political reasons 
!pattfn°l'^78 p Tsl) ' systematic information is better than none" ' 

Deciding Whether to Design Your Own Instrument 
or Select an Existing One 

Your next task, after you have chosen your research design, is to decide whether you want to 
design your own instrument or select ari existing one. When faced with designing an evaluation of 
vo.cational education, an evaiuator may consider it best to select an existing survey instrument This 
course of action does have several benefits, if the instrument has been used and fully evaluated el'^ 
where. Items wi probably be well-written and bad items eliminated or redone. Further sOrne in tru- 
ments have published "norms," which make interpretation a bit easier. • ""iJ^ner, some instru 

However, existing instruments should not be blindly selected. There are no standard instruments 
for measuring training satisfaction and job satisfaction or any other construct. There are many well- 
used instruments that have severe problems, such as poorly written items or inadequate response 
modes. Further, while published "norms" may provide a somewhat useful comparison there is no 
guarantee that the samples making up the norm are representative of the population or are even 
within the same population as the saniple, being evaluated. ■ ' 

^ Finally, the major concern is that an ex'^ing instrument may not be appropriate to the question 
being asked, a particular concern within the training-satisfaction and job-satisfaction areas There is 
- no correct hst^of dimensions of training satisfaction and job satisfaction. Furthermore, the dimensions 
chosen from the existing instrument may not precisely tap the variable felt to be affected by voca- 
tional education. For example, satisfaction with promotion opportunity is a common dimension 
However, the vocational education program may be predicted to affect satisfaction with career 
opportunity, and not necessarily satisfaction with promotion opportunity Vvithin one's current organi- 
zation (which IS more strongly affected by the availability of openings than the satisfaction with 
car-er opportunities). While the existing instrument may seem to measure the desired outcome 
a/... rately enough, even small differences such as the one pointed out here can result in inaccurate 
data collection. 

t . " ..." 

A closely related pointjs that many existing measures of job satisfaction include fairly global 
dimensions. For example, the well-known Job Descriptive Index (JDI) has many strengths and has 
been valuable in many studies, but its dimensions are fairly broad. For example, does dissatisfaction 
'With the work rnean that the respondent is. dissatisfied with the amount of challenge, responsibility 
feeling of accomplishment, variety, and enjoyment of the activities themselves? 



Summary 



Unlike research studies designed primarily to add to the body of knowledge, evaluation studies 
are designed primarily to provide information to.decisibn makers. Thus, no effort must be spared in 
increasing the probability th^t the evaluation information generated by a study will be used In 
writing the objectives of a study, determining the respondents, and choosing the appropriate research 
design including the instrument, the evaiuator should be guided, at all times, by the specific informa- 
tion needs of relevant decision makers and information users. 



CHAPTER IV 
MAILED QUESTfONNAIRE 

The review of studies on the effects of vocational education on participants showed that the 
mailed questionnaire is the most popular method of data collection (Mertens et al. 1980) The 
reason for its popularity is obvious. It can reach respondents in widely scattered areas quickly and 
at a relatively low cost. However, there are serious problems associated with its use, as will be 
discussed later. . ' . 

In this chapter, the mailed questionnaire is presented as an alternative method for collecting 
data^that measures traininoand job satisfaction of former vocational students. Discussion focuses on 
the following topics: advantages and disadvantages of this method, and techniques for increasing 
the mternakyalidity and geiieralizab|lity of data collected by mailed questionnaires. 

It is anticpated thatjnost users of this handbook will be included into the following major 
categories: those intendirfg to develop a new questionnaire, those intending to improve questionnaires 
already constructed, and/those intending to adopt or adapt questionnaires developed by others The 
principles and concepts discussed here will prove helpful to all groups. Readers belonging to the last 
two groups are encouraded,to use the checklists in determining where improveme.ns can be made 
on their instruments. I 

Some readers may be interested in examining instruments that have already been developed by 
states, local, agencies, and research institutions. These readers may refer to Vocational Education 
ii/feasures: instruments to Survey Former Students and Ttieir Empioyers (Gray,'IVfcKinhey, and 
Abram 1979). Besides the instruments, the publicaftion contains an abstract for every bited instru- 
ment along with the following information: title of instrument, developer of instrument, availability, 
instrument description, and administration. If the information was available, the abstract also 
contain data on reliability and validity. 

When to Use a Mailed Questionnaire 

A questionnaire is "a group of printed questions used to elicit information from respondents ■ 
by means of self report" (Anderson et al. 1975, p. 311 ). It is the method most widejy used in 
•descriptive research because of its clistinct advantages oyer other methods, particularly its ease in 
administration. However, it is not for everyone. There are serious limitations, as well as other - 
considerations, that an evaluator must take into account before using it. The succeeding section - 
.discusses the_advanta : — .— 
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The mailed questior^naire offers the following advantages (Anderson et al. 1975 n 311) over 
*^nbe^ "'"''"'■'"3 satisfaction of former vocational students because they 

1 . Administered at relatively low expense since they are completed by the respondents 
without need for the presence of an interviewer. 

2. Distributed to respondents quickly. 

3: Answered by respondents at their own pace. 

4. Designed to maintain the anonymity of respondents, thus reassuring them that their 
answers will not beused against them in hny way, thereby eliciting more honest 
responses than might otherwise be obtained. 

5. Standardized so that all respondents receive exactly the same printed questions to 

answer, whereas in an interview a respondent'jranswerTWlJrtflu^ 
the interviewer poses the questions. 

Disadvantagesof the Mailed Questionnaire V 

\Among the problems (Berdie and Anderson 1 974, pp. 20-22) existent within the mail 
questionnaire are the following: 

1 . Low response rate. The most obvious limitation is the danger of not receiving repre- 
sentative response. Even with a proper sampling technique/unless a high response rate 
(e.g., more than 90 percent) is achieved, the study results are not representative of what 
the results would have been if all those who received questionnaires had responded 
Kerlinger (1973 p. 414) sayis: ' k . 

Responses to mail questionnaires are generally poor. Returns of less than 
40 to 50 percent are common. Higher percentages are rare. At best, the 
researchers must content themselves with returns as low as 50 to 60 percent. 

2. Reliability and validity. Because of the nature of the questionnaires, the effective 
checking of the reliability' and validity of items and answers is limited. 

3. Question limitations. The fact that written questions are sent to subjects creates^ 
certain limitations for studies using such questionnaires. Sometimes only shallow 
questions can be asked because increasingly complex questions incur greater chances 
of misinterpretation. Those who tabulate the returned questionnaires cannot always 
be^certain that the response^will be accurately interpreted during the data analysis. 
Those who conduct the study^have no opportunity to probe deeply. into answers 
received, thereby missing data of further value. 

4. Saniple limitations. I n some studies the use of the questionnaire iS not feasible. For 
^ - ^ . J.nstarice,jtudies of sampl es including peop le who do not read must use other -m eans 

of collecting information. ' 7 



5. Completers of tfie form. One usually cannot be absolutely sure who has cortipleted 
a returned questionnaire. 

6. Item independence. Because some subjects read through the entire questionnaire before 
completing it, questions asked later in the form may influence the answers to questions 
at the beginning of the form. 

-■ " , ' . 

38 



, Due to the previously stated serious drawbacks of usirig mailed questionnaires, noted above 
some researchers refrain from using them unless other methods also are beinq used The novice ' 
in questionnaire construction and administration can be especially discouraged. Kerlinqer (1973 
p. 414) says: \ . " 

The mail questionnaire ... has serious drawbacks unless\t is used in conjunc- 
tion with other techniques. Two of these defects are possible lack of response 
.- and the inability to check the response given. These defect^ especially the 
first, are serious enough to make the mail questionnaire wor^ than useless 
except in highly sophisticated hands. 

On the other hand, Dillman (1978) claims that with his Total Design %thod (TDM) for mail and 
telephone surveys a return of 70 percent or more is not uncommon. The reaV who is determined 
to use the mailed questionnaire should study carefully Dillman's TDM to avoidthe pitfallsfacinq 
most questionnaire users. The information that follows also should be helpful.A 

, How to Increase Confidence in Data \ 
Collected by Mailed Questionnaires 

The issue of data confidence is extremely important in any data-collection method and in 
research methodology it involves the question of measurement error. A self-respecting evaluator 
piakes a primary effort to control measurement error in order to increase the "truthfulness" (relia- 
bility and validity) of the data being collected! In an effort to lessen measurement error, the following 
strategies are.discussed in this chapter: (1) determining how many items there should be per dimen- 
' ^'°!i'V;7' ^^^^'■"^'"'"9 what questions to ask, (3) determining appropriate wording of the questions 
and (4) determining appropriate response options. 



Determining How Many Items per Dimension 

An examination of follow-up instruments ussd by state and local vocational education agencies 
shows tSiat information on training satisfaction and job satisfaction are either collected as uni- 
dimensional variables (i.e., one global question) or as multidimensional variables (i.e., several ques- 
tions/. The question, therefore, is how many items per dimension? 

It is generally best to include more than one item that is intended to measure a given construct, 
particularly when that construct is central 'to the study. There are several reasons for this decision. 
Any one item includes some measurement error; that is, some of the variance in the responses across 
subjects IS due to error rather than true score. If the sources of this error can be considered to be 
random factors (such as how wording is interpreted or how the scale's anchors are defined) then 
using multiple items will increase the construct ..validity of the score. This is true because the random 
errors tend to cancel each other, while the "true sicore" components will add up. Accordingly, it is 
.often asserted that the reliability of a scale generally increases as items are added (assuming that they 
are good items). 

• • ^ 

Another reason for using multiple items is to allow for examination of the convergence among 
Items. If two items are supposed to be me: uring the same thing, then they should correlate to a 
reasonable degree. The exact number that is "reasonable" depends upon whet|jer you are criticizing 
someone else's instrument or trying to validate one you have created. 



When new itenns are being created to measure a construct, it is best to include at least three per 
construct. Even very similar items can occasionally lack convergence. If three items are used, chances 
are good that two will prove to be good items md converge at a catisfactory level. (Pretesting the 
instrument to determine convergence and requiring rework items is a good idea, of course.) 

While there are advantajges to multiple items, there are also disadvantages. As the scale gets' 
luilcltaf, the marginal utility of additional. items for Increased reliability declines. It is often difficult 
to compose several items that measure the same construct without becoming redundant. Such 
repetition can annoy respondents, who feel that they keep answering the same question (which, in 
fact, is true). Additional items obviously increase the length of the instrument and may lower the 
response rate, cause fatigue, invalidate Items near the end, and lower the number of constructs that 
can be measured. It is suggested that,.depending on the importance of the construct to the study, 
there should be at least three but no more than five items per construct. 



Determining What Questions to Ask 

i" • . 

In chapter 3 you were asked to list the specific objectives of your study in measuring the 
training satisfaction and job satisfaction of former vocational students. These objectives should 
serve as your bases in determining what questions to ask. The example that follows illustrates how 
specific objectives are translated into specific items in the questionnaire: 

Specific objective: . To determine the job satisfaction of former vocational students, 
class 1979-80, who are employed by the industries established , 
since 1975 in Brookings City 

Item for the Questionnaire: 

1 . Rate the degree to which you are satisfied with the following aspects of your present job: 

• . Neither 
\ Satisfied 

Highly ^ nor Highly 

Satisfied. Dissatisfied Dissatisfied 

(1) 12) (3) (4) (5) 

Salary . 

Fringe Benefits 

Working Conditions ^ ____ 

Status \ 

Other (specify: . . ■ 



After all the items are written, you may want to gr6up those that are similar or related. Your next 
task is to arrange your groups of items in the questionnaire so that they are presented in correct 
psychological order to the respondents. This will create a smooth flow of ideas for the respondents 
who are answering the questionnaire. They should not feel as though they are being subjected to a 
quiz or examination. As a rule of thumb, difficult questions^re placed at the end. 
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Another useful procedure for determining what questions to ask is suggested bv Selltiz 
Wrightsman, and Cook (1976, p. 543). They say: ' - 

An excellent test of one's performance in this stage of questionnaire construc- 
tion and, at the same time, a valuable aid, is the preparation of "dummy tables" 
showing the relationships that are anticipated. By drawing up such tables in 
advance, the investigators force themselves to definite decisions about what 
data are required and how they will be used [italics added] . 

Before determining whether you used the proper wording, structure, and response options in 
your questions, take a few minutes to think about the questions posed in checklist 5. It is worth 
your time to move ahead cautiously. Remember, the most troublesome errors in questionnaires 
creep in unwittingly, even in 'obviously simpfe questions' " (Selltiz, Wrightsman, and Cook 1976 
p. 547). 

Determining Appropriate Wording of the Question — 

After decisions have been made regarding the questions to ask, you are ready to address'one of 
the most important, as well as the most difficult, tasks in the entire phase of the study-writing a 
good questionnaire item. "The formulation of good questions is much more subtle and frustrating 
than IS generally believed by those who have not actually attempted it" (Goode and Hatt 1962, p. 
132). Many experts believe that the most important defect of commercial survey research is im- 
properly worded questions. 

In regard to question wording, two important, separate decisions must be made: deciding question 
structure and deciding actual choice of words. Dillman (1978, p. 86) identifies four basic types of 
question structure according to the nature of response behavior asked of the respondents: open-ended, 
close-ended with ordered choices, close-ended with unordered choices, and partially close-ended. 
Table 2 explains the uses, the advantages^ and the disadvantages of these four types of question 
structure. 

The type of question structure to use depends on the kind of information the evaluator is 
attempting to obtain in the survey. Selecting the wrong structure may mean getting the wrong infor- 
mation and receiving answers to the wrong evaluation questions. Additionally, the evaluator should ' 
consider other factors such as staff expertise, time available for the study, and financial resources. 
Analysis of results of certain structures, such as the open-ended structure, demands considerable 
expense money, a lot of time, and a high degree of expertise (Jacobs 1974, p. 10). 

Whichevetv structure is used, the key issue is fb maximize the specificity of 'the item. The object 
being referred to should be concise and clear, particularly when the effects of vocational education 
are being evaluated. As argued in the preceding section, any effects are likely to be very specific and 
will not be identified by one global satisfaction question. 

After a decision is made on question structure, the evaluator needs to decide on the actual 
choice of words. 'The wrong choice of words can create any number of problems-from excessive 
vagueness to too much precision, from being misurt^rstood . ..^from being too objectionable to 
being too uninteresting and irrelevant" (Dillman 1978, p. 95). Payne (1951, p.*9) adds: 

Question wording invovles more .than toying with this word or that to see 
what may happen, however It is more than a mere matter of manipulation 
of words to produce surprising illusions. The most critical need for attention 




^ CHECKLIST 5 
Decisions Aboiit Question Content 



Instructions: Examine each questioi^ in youi^ questionnaire in terms of the follojj/ing points: 



. / 



Yes 


No 




□ 


□ 


1. 


□ 


□ 


2. 


□ 


□ 


3. 


□ 




4. 


□ 


□ 


5. 




□ ■ 


6. 


□ 


□ 


7. 



1. Is this question necessary? Just'how will it be useful? 

■. /.. 



"idents ha/e 



/ 



personal ex^'erience?^ 



Is the question^ content sufficiently general and free from too muih 
cOncreteness artd specificity? { 



accompanying qu^estions to Isa lance the emphasis? 
7. Will the respondents giye the information that is asl<ed for? 



Note: Ideas in this checklist were tal<en from C. Selltiz,.L.S. Wrightsman, and S W Cook 
Research Methods in Social Relations (New York, New York: Holt, Rinehsrt and Winston, 1976). 



lABLfc 2r- 

Types of Question Structure 



1. Open-^nded Questions 

Examples 



• In your opinion\ what is the most satisfying part of your vocationa! trainingT 

• What specific things about this vocational program have you found to be least 
useful to you in your present iob? 



Advantages 



Disadvantages 



^ Lend themselves to situations in which respondents can express themselves freely 
and/or recall a precise piece of information without difficulty. 

• Are useful when researchers cannot anticipate the various ways in which people 
are likely to respond to a question. 

• Tend to stimulate free thought, be suggestive, probe people's memorie*' ^nH 
clarify positions. \ 

• Can bevery demanding. The tasks of articulating answers is difficult for most 
respondents, especially for those with low educational attainment and for those 
who lack experience in communicating ideas to other people. 

• Are time-consuming for the respondents and may affect the response rate. 

• Elicit answers difficult to code and summarize. . 



2. Close-ended Questions with Ordered Answer Choices 



Example 



• Considering your vocational training, rate the degree to which you are 
satisfied with each of the following: 





Satisfied 


Not Sure 


Dissatisfied 


Methods of instruction 


□ 






Facilities and equipment 


□ 


□ 




Guidance service 


□ 


□ 


□ . 


Placement services 


■ □ 


□ 


□ 


Cooperative worl< experience 






□ 


Apprenticeship training 









Advantages 



Disadvantages 



• Are suitable for determining such .things as intensity of feeling, degree of Involve- 
ment, and frequency of participation. . ^ 

o. Elic|t responses suited to many forms of statistical analyses. > 

• Place little desviand on respondents. 

• Tehd to be very specific, causing respondents to think about a limited aspect of 
life in a limited way. Having a response dimension narrow in scope enables 
respondents to place themselves at the most appropriate point on a scale implied , 
by the answer choices. Only appropriate If the researcher has a well-defined issue. 
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Table 2 (continyed) 



3. Close*ended Questions with Unordered Answer Choices 



Example 



Which one of the following aspects of your vocational training are you most 
satisfied with? (Check one) 

□ Methods of instruction 

□ Facilities and equipment 

□ Guidance service 

□ Placement services 

□ Cooperative work experience - 

□ Apprenticeship training 



Advantages 



Disadvantages 



• Are useful for establishing priorities among issues and deciding among 
alternative policies. 

6 Do not limit respondents to choosing among'gradations of a single concept 
Each choice is an independent alternative representing a different concept. 

f .Are generally more difficult to answer than those containing ordered answer 
choices, inasmuch as respondents must often balance several ideas in their 
nds at the same time. 




lude obtaining useful results unless the researcher's knowledge of the subject 
►ws meaningful choices to be stated; possibly eliminates the most preferable 
option of the respondents. 



4. Partially Close-ended Questions 



Example 



• Which one of the following aspects 'of your vocational training are you 
■ most satisfied with? (Check one) 

□ - Methods.of instruction 

□ Facilities and equipment . 

□ Guidance service 

□ Placement services . " • 

□ Cooperative work experience 

□ Apprenticeship training ^ 

□ Other (specify:,, ^_ 



Advantages 
Disadvantages 



• Allow building of variables and testing of hypotheses. 

• Preclude forcing respondents into boxes in which they clearly do not fit. 

• Seldom obtain sufficient number of additional responses in the open-ended 
option. 



Note: In making this table, some ideas were taken from D. Dillman, Mail and Telephone Surveys: 
TpA jo^3l^^^i gn Met liod (New York, New York: John Wiley & Sons, 1979). 
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to Wording is to make sure that the particular issue which the questionnaire 
has in mind is the^particular issue on which the respondent gives his^ 
answers . . . , 

To assure that the intended issue is understood, then, is the fundamental 
function of question wording. . 

The goal, then, is to word carefully and clearly each question so that all respondents interpret 
It in the same manner as the question designers. "Generally, the most effective questions are worded 
as simply as possible" (Berdie and Anderson 1 974, p. 39). Table 3 shows some "don'ts" in question 
wording. Checklist 6 is designed to help you determine appropriate question wording. 



Determining Appropriate Response Options 

. A well-constructed questionnaire includes both proper wording of questions and proper response 
options. Confusing options^'increase measurement error (unreliable results) and the percentage of 
nonresponse rate. The following suggestions (Berdie and Anderson 1974, pp. 45-47) and examples 
are offered to help you design appropriate response options for questionnaire items. 

1. Make certain that one response category is listed for every conceivable answer. To omit an 
option forces people to answer in a way that does not accurately reflect reality. 

Example: In your opinion,'what is the most satisfying part of your present job? 
(check one) ^ 



Poor Options (very few choices) 

^ Salary 

____ Fringe benefits 



Better Options (more choices) 

Salary 

Frmge benefits 

. Working conditions 

^ Coworkers 

Others (please specify: 



.) 



Include a "don't know" response option whenever respondents may be unable to answer. 
Although a "don't know" option may be viewed as offering respondents an "easy out," it is 
probably better to include this option than to take the chance of obtaining inaccurate 
information by forcing people to respond to an item about which they know,nothing. 

Example: In your opinion, what aspect of your vocational training needs the most 
improvement? (check one) 



Poor Options 

Facilities and equipment 

v_ Teachers 

Apprenticeships/coop 



Better Options 

Facilities and equipment 

Teachers 

Apprenticeships/coop^ 

Others (please specify: 

Don't know 
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TABLES 
Some Don'ts fri Question Wording 



Ambiguity 



Misperception 



I- 

I 

Loading 



Special 
Wording 
Problems 



• Avoid questiQnsthat are incomplete^ imprecise, or indefinite. 

• An incomplete question.is likely to lead to confusion. ' 

• An imprecise question conveys an unreal meaning or suggests an r 
inaccurate answer. 

• An indefinite question contains hazy words-e.g.> "frequently " 
"usually/' "often/' "always/' etc. - . 

• Avoid questions containing words that lie outside the respondents' 
experiences and have no meaning to them. 

• Avoid questions containing words so familiar to respondents that they 
may be confusid with similar-sounding vyords. 

• Avoid questions that violate local idioms. When a question is worded 
contrary to expectations/ respondents are likely to respond, nonetheless, 
in terms of their expectations. , 

• A question is loaded when something in it suggests to the respondents 
that one particular response is more desirable than another. /i 

• A question is loaded when it provides unfair alternatives. 1 

• A question is loaded when it contains emotionally charged words or 
stereotypes. ; 

• A question is loaded when it is embarrassing. 

• Avoid questions that assume too much knowledge on the part of the 

respondents.v 

• Avoid lengthy questions, such as two-part questions. 

• Avoid use of double negatives. 

® Avoid illogical sentence construction. . 



fl?^®;^''^^' this table were taken from C. H. Backstrom and G. D. Hu'rsh, Su\ey Research 
(Northwestern University Press, Chicago, Illinois, 1964). \ 



CHECKLISTS 
Decisions About Question Wording 



Instructions: Examine each question carefully in your questionnaire in terms of the 
following pomts: 



Yes 


No 






□ 


n 


1. 


Can the question be misunderstood? 




I ■ i 




Does it contain unfamiliar or unclear phrases? 


"□ 


□ 


3. 


Does the question adequately express the alternatives with respect 
to the point? 




□ 


4. 


Is the question misleading because of unstated assumptions or 
unseen implications? 




n 


5. 


Is the wording biased? 


□ 


n . 


6. 


Is it emotionally loaded or slanted toward a particular kind of answer? 


□ 


n 


7. 


Is the question wording likely to be objectionable to the respondent 
in any way? 




□ ■ 


. 8. 


Would a more personalized or less personalized wording of the question 
produce better results? 


□ 


n ■ 


9. 


Can the question be better asked in a more direct or a more indirect form? 



Note: Ideas in making this checklist were taken from C. Selltiz, L. S. Wrightsman, and S W 
Cook Research Methods In Soc/a/ Relations. (New York, New*York: Holt Rinehart ' 
and Winston, 1976). • 
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3. Make response options mutually exclusive and independent. 



Example: [n^Vour opinion, what is the most satisfying part of your present Job? 



Poor Options \ 
^ (options not mutually exclusive) 

_ Salary 

- Medical benefits 

Fringe benefits 

Working conditbns 

Status 



BettenOptiohs 

(options mutually exclusive) 

^ Salary 

Medical benefits 

Other fringe benefits 

^ Working conditions 

. Status 

— — Others (please specify: 



J 



on'S rdl^ptS" '"^'""^ ""-^^ "' options 

Example: ^ Please rate the degree to which you are satisfied with the following: 
Poor Scale (unequal number of options on each side of the middle position) 



Satisfied 
(1) 



Neither 
SatisfiiSd 
nor 

Dissatisfied Dissatisfsed 



(2) 



Salary 

fringe benefits 
Working conditions 
Status 

Others (please specify: 



(3) 



Highly 
Dissatisfied 

(4) 



) 



Better Scale (equal number of options on each side of the middle position) 



Highly 
Satisfied 

(1) 



Satisfied 
(2) , 



Salary 

Fringe benefits 
Working conditions 
Status 

Others (please specify: 



Neither 
Satisfied 
nor 
Dissatisfied 

(3) 



Dissatisfied 
(4) 



Highly 
Dissatisfied 

(5) 



_) 
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5. Make sure to label the midpoint according to the "exact" meaning the scales require. 
Example: Please rate the degree to which you are satisfied with the following: 
Poor Scale (midpoint not labeled) ' 



Highly 
Satisfied 

. (1) 



.Satisfied 
(2) 



(3) 



Dissatisfied 
(4). 



Highly 
Dissatisfied 

(5) 



Salary 

Fringis benefits 
Working conditions ' 
Status 

Others (please specify: 



, ) 

Better Scale (midpoint labeled according to the exact meaning): 

, Neither 
Satisfied 

Highly nor 
Satisfied Satisfied Dissatisfied 

, . (1) ' (2) (3) 

.Salary . _. ^ • 

Fringe benefits ' ' • " - 

Working conditions :• ^ 

Status ___ ____ 

Others (please specify: ^ ' 



Dissatisfied 
(4) 



Highly 
Dissatisfied 

(5) 



_) . / - 

6. Arrange responses vertically. 

Example: In your opinion, what aspects of your vocational training needs the most 
irpprovement? (check one) , 

Poor Response Arrangement (horizontally arranged) 

Facilities and equipment - Teachers , 

Apprenticeships/coop ' Others 

Don't know 

^. . .■ 

Setter Response Arrangement (venlcaWy arranged) 

Facilities and equipment 

Teachers 

Apprenticeships/coop 

Others 

Don't know 
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7. Make certain the respondents know exactly what information they should put in the 
> blanksoffill-in-the-blank items. 

Poor Direction: Better Direction: 

Age in years years of age at last birthday 

; Now you are probably ready to write your response options. After you have written them and 
before you begin to read the next section, take a few moments to review your response options by " 
followmg the mstructions in checklist 7. r- r- y 

Pretesting ^ . 

The reader is forewarned that good question construction required a deliberate process entailing - 
considerable investment of time and material resources. It can be a frustrating pro^riss for evaluators^ 
especially if they need to meet short deadlines. Oftentimes, it involves pretesting to reveal the 
questionnaire's weaknesses. LundL >g (1942, p. 1S8) warns: 

The inexperienced researcher is likely Xo be impatient with this praliminary 

work, which may seem lii^e hair splitting over the meanrng of word3, and other 
details. But patience and care in this preliminary work may make all the 
difference between success or failure, both in the cooperatioii of the respondents 
and in the reliability and validity of the results. 

Pretesting can be informal or formal. The most important aspect of the exercise is to pretest the 
questionnaire with respondents representative of the group of former vocational students who will 
eventually receive it. If the evaluator desires, pretesting can generate data that will reveal the read- 
ability, reliability, and validity of the instrument. (For additional discussion on the subject, see 
Franchak and Spirer 1 978 and McCaslin and Walker 1 979.) 

How to Increase the Generalizability of Data 
Coliected by Mailed Questionnaires 

As mentioned earlier, one of the most serious problems of using the mailed questionnaire is low 
response rate. Partial returns "may introduce bias that will render the obtained data useless" (Van 
Dalen 1973, p. 325). Inadequate response is especially critical if questionnaires have been sent to a 
sample, because the resulting summarized data may not represent the true response of all the target 
respondents. As a result, generalizability of the data collected and their usefulness for decision making 
and program improvement are impaired. 

This section details some valuable strategies for increasing the response rate of mailed question- — 
naires. Strategies are focused on preventive tactics; that is, the removal of possible causes within the 
evaluator's control that prevent the respondents from answering and returning the questionnaires. 
Included are proper format considerations and other strategies for stimulating response. 

Format Considerations 

Proper formulation of the instrument is a critical phase of questionnaire development. Improper 
format not only creates problems for data coders and tabulators, but can also lead to misinterpretation 
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CHECKLiST 7 
Deciding about Response Options 



Instructions: Respond to the following questions as appropriate: 










1. What type of response options are present in your mailed questionnaire? 

" 1. 








2. 










3. . 






• 




4. 










J .-I 

2. If a checklist is used, 










1. Does it cover adequately all the significant alternatives 
without overlapping? 


□ 


Yes 


□ 


No \ 


2. Is it of reasonable length? 




Yes 


□ 


isio 


3. Is the wording of items impartial and balanced? 


□ 


Yes 


□ 


No 


3. Is the forrh of response — 










1. Easy? 


□ 


Yes 


□ 


No 


2. Definite? 


□ 


Yes 


□ 


No , 


, 3. Uniform? 


□ 


Yes 


□ 


No 


4. Adequate for the purpose? 


■ □ 


Yes 


D 


No 
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of questions (thus increasing measurement error) and low response rates (thus weakening the 
generahzability of results). Careful consideration must be given to two areas when format decisions 
are made: 

1. The respondents (former vocational education students). The format should enable 
them to read and answer questions as easily as possible. Keep in mind that completing 
a questionnaire is an imposition. 

2. Data coders and tabulators. The format should allow easy data coding and tabulation. 

Goode and Hatt (1962, p. 43) succinctly sunimarize the important principles to observe in ' 
formulating a questionnaire: 

Common sense dictates pertain practices about the design of the, mail question- 
naire. The mail questionnaire should be attractive and easy to fill out, have 
adequate space for response, and be legible. A neat, well-organized; attractive 
questionnaire should increase the response rate. This assumes that people 
associate appearance with quality and are more willing to complete and return 
the'^form. Conversely, a sloppy, crowded, or poorly reproduced questionnaire 
will have adverse effect CO response rates. 

Checklist 8 will help you review and improve the 'ormat of your mail questionnaire. 



Strategies to Stimulate Response 

Following up nonrespondents is a difficult and costly process. It is, therefore, important to 
exhau^ allmeans to keep the percentage of nonresponse as low as possible. The emphasis should be 
on preventive tactics; i.e., the employment of strategies prior to the receipt of the questionnaire by 
the respondents. 

Do you know your respondents? Have you anticipated all conceivable objections to their 
answering and returning the questionnaire? These questions are crucial in devising specific strategies 
to stimulate response. Your tactics should be tailored to your specific respondents, former vocational 
students, and should include techniques appropriate for this particular group. 

The ultimate objective is to obtain as many responses as possible, in the form of 
completed questionnaires, which provide usable data. If questionnaire forms 
meet criteria of physical attractiveness and obvious consideration for the 
respondent, it is believed that the percentage of replies will be sufficiently high 
to fulfill the requirements of the investigator. Every conceivable inducement 
should be used in the hope of convincing one more potential respondent to take 
the time and effort necessary to answer the questionnaire (Nixon 1954, p. 486). 

Inducement for the respondents to reply can include making precontact either by telephone or 
mail, preferably by someone who is known by the respondents (e.g., former vocational teachers or 
guidance counselors). In addition, the evaluator may use material or monetary inducements such as 
sending cash or small gift items such as pencils, school decals, or buttons, or evaluators could use a 
raffle as an inducement with the respondents included in a raffle if they r3turn their questionnaires. 
Some social researchers find that such inducements do increase significantly the rate of response. 

. \ 

Another strategy for increasing response rate is to start an early campaign to inform your 
target population. Some local schools begin their information drive while the students are in their 
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CHECKLIST 8 
Format Considerations 



Instructions: 


Examine your questionnaire format in terms nf thp fniinuumn r>nne;rJa.-o*;«r,«. 

/ '«i*-*'»*t.i wi ii luii w iwiiiiaLiii i.ciiii;>ui iric loiiowiny consiQBrations: 


Yes 


No 




D 


D 


1. Is the questionnaire "appealing to the eye" and as easy to complete as possible? 


, □ 


D 


- 2. Did you number questionnaire items and pages so the respondent will not become 
confused while completing'the form? 


D 


□ 


3. Did you put an identifying mark on each page of the form so that if one page 

SnOuICi OPt QPnaratoH frnm tho roc* !♦ ^^^.a^^ l 

oii^uiu yul :>(;pdrdLt;u Trom in 6 re ST, It can be reattached? 




□ . 


4. Did you put the name and address of the person to whom the form should be 
returned at the beginning and end of the questionnaire even if you included 
a self-addressed envelope, since questionnaires are often separated from the cover 
letter and the envelope? 


□ 


□ 


5. -Did you put the study title in bold type on the first page of the questionnaire? 


□ 




6. Did you include brief and clear instructions (preferably bold or italics) for 
uuiiipiexing ine Torm ana aaaitional cla/itication and examples before sections 
that may be confusing? 


□ 




7. «s the question led up to in a natural way, i.e., is it in correct psychological order? 


□ 


□ 


f\ 1 IlfH \/Oii no/tin tm^i^n n fm^i . • • . \ 

,o. uio you pegin with a few interestipg nonthreatenmg questions because 
introductory questions that are either threatening or dull may reduce the 
likelihood of the subject's completing the questionnaire? 


□ 


□ . 


9. Did you avoid putting important items at the end of a long questionnaire? 




. □ • 


10. If questions appear on both sides of the page, did you put the word "over" on 
the bottom of the front side of that page? 


□ 


□ 


1 1. Did you try to make smooth transition between sections so that the respondent 
does not feel he is answering a series of unrelated "quiz" questions? 



/Vofe; Ideas in this checklist were taken from Douglas R. Berdie and John F Anderson 

5ofk M ' v^-J^u^'!'"2"^'!"^ '^^'^'''^ Methods in Social Relations (New 

York, New York: Holt, Rinehart, and Winston, 1976). :. 
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senior ywr. They are made aware of the objectives and importance of the study and their role in 
iu% gradu/jtmg, the students know already that they will be participating in a follow-up 

V 

Now, let us pause and take a hard look at your completed questionnaire. Checklist 9 is desiqned 
to help you review the strategies that you have selected to stimulate response. Remember "An ounce 
of prevention is better than a pound of cure." 

Dealing with Nonres,3onse Bias 

If only a certain percentage of your sample completed and returned the questidnnaires would ' 
the summarized response reflect the entire sample? Your answer may be yes, or no. There is'no 
simple and easy answer to this question. 

Questionnaire users must consider the response rate problem as it uniquel^ 
applies to their own situation. Careful consideration of the situation should 

dictate whether or not [nonresponse] bias is a danger and, if so, what 
decisions must be made about the percentage of response needed to tensure] 

representativeness and what should be done with the data if this level of 
response is not reached (Berdie and Anderson 1 974, p. 51 ). 

The following strategies (see also Franchak and Spirer 1978) may help you deal with the 
problem of nonresponse bias: 

1 . Compare the respondents and nonrespondents on certain variables. If the two groups are 
similar perhaps you may ignore the nonresponse issue. Hoyvever, keep in mind that com- 
paring respondents to nonrespondents only on demographic data may not be relevant to 
the response rate problem. You are concerned with attitudinal data, and "the fact that 
nonrespondents may or may not share certain demographic'features with . . . respondents 
often says very little as to whether or not they share similar attitudes" (Berdie and 
Anderson 1974, p. 50). 

2. Get a random sample of nonrespondents and follow up this sample. Consider the responses 
of this Sample as the responses of all nonrespondents. This is considered a better alternative 
than the first one; however, the problem is that some may not respond at all. 

At this point another word of caution is in order. You may be tempted to increase your sample 
size to offset nonresponse. For example, instead of sending questionnaires to a randoifi sample of 100 
as originally designed, questionnaires may be sent to a random sample of 1^0 former vocational 
students. If the return rate is 80 percent, you will receive 96 completed returns, which is close to 
100. While in some cases, increasing the sample size may serve a useful purpose, it does not solve 
the response rate problem, which is the^^ercentage of respondents. It is of extreme importance that 
the evaluator place emphasis on increasin^^e response rate, not the^mple size. 

\ . ■ ' 

Summary 

While the mailed questionnaire is the most popular method of collecting data, it is fraught with 
serious limitations (see figure 3). Low response rate and the difficulty of checking the accuracy of 
the responses are the two most important problems. To ensure greater confidence in the data collected 
by mailed questionnaire and also better generalizability of the data, certain methodological controls 
and strategies need to be instituted by the evaluator. Figure 4 illustrates some techniques for decreasing 
measurement error, and figure 5 shows some techniques for increasing generalizability. 
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CHECKLISTS 
Stimulating Response 



Instructions: Take time to review your strategies to stimulate response by answering the 
following questions: 

1. How will you relate (i.e., formally or informally) to your sample of former vocational students? 



2. Hovv will you be able to guarantee respondent's anonymity or confidentiality? 



^ mann^rT '^D Yes°" N P""'"^^^ "i°st professional and appealing 

4. Have you^refully considered the content end approach of your preietter and cover letter? 
Li Yes □ No 



5. Have you considered offering some type of incentive (such as giving the respondent small cash 
or gift Items) to encourage responses? □ Yes □ No 

6. Haveyou identified sufficient resources from which to obtain updated addresses of people in 
— — ytJilr study? □ Yes □ No 

7. Where is the best place to send the questionnaire to the respondents? □ work? □ home? 
□ both work and home? 

8. Have you considered using "high-powered" mailing tactics (such as the use of special delivery 
certified, or first class mail) to stimulate responses? □ Yes □ No 

9. What methods will you employ to determine why people are not rsturnina completed ques- 
questionnaires? - . 



10. Have you allocated sufficient resources to follow-up nonrespondents? □ Yes D 



No 
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Checklist 9 (continued) 



11. What nr.ethod have you considered using for follow-ups? 




/ 

Note: Some ideas in making this checklist were taken from Douglas R. Berdie and John F. 
Anderson, Questionnaires: Design and Use (Metuchen, N.J., The Scarecrow Press Jnc 
1974). / 
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FIGURES 

Mailed Questionnaire: 
Advantages and Disadvantages 



Mailed Questionnaire: 

A group of printed questions 
used to elicit information 
from respondents by means 
of self-report. 



Should I us ejt? 



Considerations 



Advantages 

• Relatively low a^it 

• Ease in administration 

• Ease in standardization 

• Ease in tabulation 

• Suitability for large sample 
« Respondents' anonymity assured 



Disadvantages 

• Low response rate 

• Problems of validity and 
reliability 

« Question limitations 

• Sample limitations 

• Item independence 
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FIGURE 4 

Decreasing Measurement Error 
of Mailed Questionnaire 



Goal: Increasihjg reliability 
of data collected by I 
questionnaire. F 



\ 



X- 



Greaterlionfidence in the ) 
utility of evaluation data 
for decision maki ng f' 



> 




Decrease measurement^ 
error by 




1. Determining proper questions 
to ask (seeV}bjectivGS of the study) 



Z Determining proper wording for 
questions simply so each item ' 
is open to one interpretation only./ 




3. Determining proper response!- 
options. j 
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FIGURES 

Increasing Data Generalizahility 
of Mailed Questionnaire 



Increasing the generalizability t 
of data collect ed b y questionnaire.^ 



Greater utility of evaluation 
data in decision making. 



Increase the external | , . 
data-validity by: 1 • 



4^ 



Good format. Make it easy \ 
for respondents to ansvi/er ' 
questions as well as for the 
data coders and tabulators 
to use the inf ormatio^i. 



Employment of varied techniques. ^ 
for stimulating response. f 



A. 



Employment of proper 
techniques for handling 
non response. I 
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CHAPTER V 
INTERVIEWING 



In addition to the use of the mailed questionnaire, interviewing is a common form of collecting 
data employed in survey research. Data are collected by talking with people, but interviewing entails 
more than asking questions of people (Kester 1979, p. 32). While asking questions is a natural part 
of the process, interviewing requires specialized skills and precise procedures to ensure full and 
accurate data collection (Institute for Social Research 1976, p. 1 ). Thorough and careful planning is 
required. As Jn the use of the questionnaire, there is a need for instituting proper methodological 
control to ensure a high degree of data confidence and gei^eralizability. 

This chapter explains the qse of the interview as an alternative method for collecting data to 
measure training and job satisfaction of former vocational students. The concepts discussed comple- 
ment those cited in another National Center publication. The Case Study Method: Guidelines and 
Practices for Vocational Education (Spirer 1980). The chapter focuses on topics that have direct 
bearing on the issues of internal data validity and generalizability. Specifically, the topics that are 
discussed are (1 ) selection of the interview and a data collection method, (2) strategies for increasing 
internal validity of confidence in the data, and (3) strategies for increasing the generalizability of 
interview data. 



, When to Use the Interview 

Before the interview is selected as a data collection method, careful consideration should be 
made as to its strengths and weaknesses in relation to tlie evaluation goals; the population to be 
surveyed, and the precise survey objectives. The following listing itemizes common advantages and 
disadvantages (Anderson et aL 1975, p. 216) of using the interview instead of the mail questionnaire: 

Advantages: 

1 . Does not require reading or writing by respondent 

2. Is adaptable to unforeseen circumstances 

3. Provides an opportunity to obtain a representative sample ofrespondents 

4. Permits study of complex and sensitive topics 

5. Ensures that respondents understand questions 

6. Permits the respondent's environment to be structured to some degree 

Disadvantages: 

1. Requires trained interviewers 

2. Is subject to variable and unpredictable interviewer bias 

3. Tends to be difficult to structure because of large response volume 

4. High cost involved 
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Perhaps the most important drawback in using the interview as a data collection method i/ 
the cost, especially for face-to-face interviews. However, for certain respondents, such as special 
population groups, it may be the most feasible method. For example, it may be unwise to us^ th- 
questionnaire in collecting data from individuals who may have difficulty reading and undefstandinq 
written questions, and writing responses. In this situation, an interview may be more app/^priate. 

If you decide to use the interview, another decision involves the specific type to IC Many 
other factors, such as obtaining a representative sample, deciding on questionnaire des'igsi," obtaining 
accurate answers and fulfilling administrative requirements need to be considered (Tillman 1978 
pp. 74-75). In the above example, it may be appropriate to use a face-to-face interview if the sample 
IS located in a specific geographical area such as a town. If the sample is distributeti throughout the 
United States, the use of the telephone interview may be more appropriate. Checklist 10 is designed 
to help you make a proper choics in selecting the type of interview to employ iii measuring the 
training and job satisfaction ot former vocational students ■ / 

■.. . ■ / 

How to Increase Confidence in the Data ^ 

Collected by Interview i •""'"^ 

As emphasized in the previous chapter, exercising methodological control for reducing measure- 
ment error should be a continuing, important concern for bo.ii the evaluator and the researcher 
Data reliability-a major determinant of data usability-is very much a function of methodological 
rigor Additionally, wlien the interview schedule is finalized, much investment /time and money) 
has been made on the study. Thus, no effort must be spared in ensuring that complete, accurate data 
are collected. Preliminary computer processing and data analysis for initial results can be undertaken 
to detect certain errors. These preliminary efforts may minimize the study cost and time of the study. 

In using the interview, the evaluator needs to pay particular attention to measurement error 
from two vantage points: question construction and interviewing process. The principles and tech- 
niques discussed in the previous chapter regarding determination of question content,' structure and 
wording apply also in formulating questions for the interview. Although interviewers have greater 
flexibility in asking questions than the mailed questionnaire-e.g., use of operf-ended questions or use 
of probes-the basic principles of question construction and the factors to consider in making decisions 
remain the same. Questions should be clear and unbiased with each one focused on a single thought 
or issue. To the extent possible, the language and syntax of the questions should correspond to that 
of the respondent. 

On the other hand, the best constructed questions will not be of much valua in collecting valid 
and useful data when posed by unskilled or biased interviewers. Though many hours may have been 
spent in perfecting the interview questions, in the final analysis the utility and effectiveness of a ' 
questionnaire in getting valid and useful information depend upon the interviewer. The following 
subsection details some strategies arid techniques for reducing data distortion during the interviewing 
process and includes some useful techniques for probing answers, 

Using'the Questionnaire In Interviewing 

Collection of accurate and useful information is the goal in using the interview schedule, which 
is the interviewer's guide in posing the questions to the'respondents. It should be impressed upon 
interviewers that "each question has been carefully pretested to express the precise meaning desired. 
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CHECKLIST 10 






Deciding the Type of Interview to Use 




Instructions: The evaluator should not be confined to one data collection methodology, but 
should be able to select the most appropriate methodology demanded by the 
situation. Take a few minutes to answer each of the items listed below: 




' Number Geographical Method Selected 
fype of Respondents Respondents Distribution (check one item) . 




. . Face-tO"Face . 


Telephone 


1. 


Regular student □ 


n 


2. 


Wopien ' " ' rn 

I 1 


1 1 


3. 


Handicapped q 




4. 


Di!?advantaged ' 


□ 


5. 


Minority q 


□ 


6. 


Limited English Proficiency □ 


□ 


7. 


Returning Adults □ 


□ 


8. 


Others (specify:) >i ^ 


□ 






1 



/ 
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in as simple a manner as possible-even a slight rewording of the question can so change the stimulus 
as to provoke answers in a different frame of reference or bias the response" (Selltiz, Wrightsman, 
and Cook 1976, p. 564). To this effect, Backstrom and Hursh (1963, p. 139) offer the followinq 
suggestions: , 

1. Always follow instructions carefully, 

2. Always study the questionnaire until you are familiar with all the questions. 

3. Always use the brief introductory approach written into the questionnaire. 

4. Always be completely neutral, informal, conscientious. 

5. Always read questions just as they are written. 

6. Always ask all of the questions. 

7. Always ask que.<,tions in the order they appear. 

8. Always record comments accurately. 

9. Always interview only the proper person . . . designated by your procedure. 

10. Always check each questionnaire to make sure you have completed every item. 

1 1 . Always interview people you don't know, and interview them alone. 

However, Dexter (1979, p. 23) cautions that there is no set of universal rules that govern all 
situations in interviewing. In the ultimate analysis, "every suggestion about how to conduct interviews 
must depend upon these all-important variables: the personality and skill of the interviewer the 
attitudes and orientation of the interviewee/and the definition by both (and often by significant 
others) of the situation" (Dexter 1979, p. 24). 

Probing and Other IntervievWng Techniques 

It is relatively easy for interviewers to ask questions and record answers. However, obtaining a 
specific, complete response, which is crucial to the interviewing process, requires skill, it is not 
uncommon for some respondents to answer "Don't know" in order to avoid thinking about a 
question, misinterpreting the question, or contradicting themselves. In all these critical cases the 
interviewer can use probing techniques. ' •' 

Probing is a technique that motivates respondents to communicate fully so that they enlarge on 
clarify, or explain the reasons behind what they have said. At the same time, it helps the respondents' 
focus on the specific content of the interview so that irrelevant and unnecessary information can be 
avoided (Institute for Social Research 1976, p. 15). A good interviewer, therefore, is alert in detecting 
incomplete or vague answers and skillfully uses neutral probes to elicit complete, specific information 
from the respondents. This technique requires the interviewers to understand the objective bf every 
question. It is only through "complete understanding of the question that the ini;erviewer can recognize 
when and where probes are needed and use them effectively" (Institute for SociyI Research 1976 p 
16 . Toward this end, several useful "techniques for probing (Institute for Social Research 1976 pp 15- 
16) are suggested: . 

1 . Repeating the question. When respondents do not seem to understand the question, 
when they misinterpret it, when they seem unable to make u(3 their minds, or when ' 
they stray from the subject, the most useful technique is to repeat the question just as 
/ it is written in the questionnaire.' 



/ 
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^^'"^u" ^^P^^^?"^ P^^s^- The simplest way to convey to respondents that you know 
they have begun to answer the question, but that you feel they have more to say is 
to be silent. The paust-often accompanied by an expectant look or a nod of the' head- 
gives the respondents time to gather their thoughts. 

3. Repeating the respondent's reply. Simply repeating what the respondents have said 
as soon as they have stopped talking is often an excellent probe. This repetition should 
be made as you are writing, so that you are actually repeating the respondents' reply 
and recording it at the same time. Be sure also that you are understood correctly. 

Using neutral questions or comments. Neutral questions or comments are frequently 
used to obtain clearer and fuller responses. The following are the most commonly used 
probes: ' 

• Let me repeat the question. 

• Anything else? 

• Any other reason? 

• Any others? 

• What do you mean? 

• Could you tell me more about your thinking on that? 

• Would you tell me what you have in mind? 

• Why do you feel that way? 

• Which would be closer to the way you feel? 

>^ff(ing further clarification. In probing, you will sometimes find it useful to appear 
slightly bewildered by the respondents', answer^!. For example; "I'm not quite sure I 
know what/ou mean by that-could you tell ...e a little more?" This technique however 
should not/be overplayed. ^ - . • » 



Dealing with Interviewer's Bias 



Interviewer's bias-"systematic differences from interviewer to interviewer or, occasionally 
systematic errors on the part of n^dny or even all interviewers"-affects the validity of evaluation data 
collected through interview (Selltiz, Wrightsman, and Cook 1976, p. 570). Personality and demo- 
graphic characteristics of the interviewers and situational factors may influence the responses of the 
respondents (Van Dalen 1973, pp. 329-330). Thus different interviewers will not always elicit the 
same response even from equivalent groups of respondents. Interviewer's bias is, therefore a reality 
of which every evaluator needs to be aware. The most common techniques for reducing in'tervit^wer's 
bias include the following: >. wci a 

1 . f'roper selection of interviewers. Common sources of bias sources are the interviewers' 
preconceived ideas or perceptions or the situation. If the result of the survey is a 
possible threat to the interviewers in any way, e.g., posing a threat to persona! interests 
or beliefs, interviewers are likely to introduce bias (Selltiz, Wrightsman, and Cook 1976 
p. 572). A good evaluator should take into account the foregoing consideration when 
selecting interviewers. Now take a few minutes of your time to complete checklist 1 1 
to help you in the selection of interviewers. 

2. Standardization of the interview. This procedu refine I udes use of standard wcrdinci in \ 
interview questions and standard instructions on probing procedure, classification of ' 
doubtful answers-suggestions all aimed at minimizing interviewer's bias. 
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CHECKLIST 11 
Selection of Interviewers 



Instruction: Please respond to the followmg questions: 
f." How many interviewers do you need? 

2. What special qualifications are required? Piaase list them. 

1. 

2. 

3. 

4. ^ ^_ 

5. 

3. What personal characteristics are required? Please list them. 

1. . ■ 

2. 

3. 

4. ' 




3. Appropriate training of interviewers. Untrained or improperly trained interviewers are 
certain to manifest greater bias than those adequately trained. The training procedure 
should include the following experiences: how to make initial contact, how to secure 
the interview, how to use the questionnaire, how to make use of probing and other 
interviewing techniques, and how to record and edit the interview. Nfeedless to say 
interviewers need the objective of every question, and the possible source.9 of data ' 
7Jq°2'°"' '"^'"<^'"9 ways of dealing with this distortion. Additionally, Van Dalen 
(1973), p. 330) recommends that the interviewer be kept ignorant of both the 
hypotheses being tested and the data returns. Knowledge of such information may 
create bias, thus Influencing the manner in which the interviewers conduct the interview.. 

4. Proper motivation and close supervision. Horror stories on data being "manufactured" 
by interviewers m the comfort of their motel rooms are not uncommon Such an 
eventuality can be avoided if interviewers are properly supervised and made to under- 
stand the importonce of getting complete, accurate information. The evaluator should 
be aware of the possibilities of bias ax various points in the data collection orocess and 
mstitute proper safeguards of minimizing it (Selltiz, Wrightsman, and Cook 1 976 p 
572). Whenever resources permit, it is suggested that a field visit be made at least 'two 
times during the interview period-a visit after one-third of the respondents are inter- 
viewed and another visjt after two-thirds of the respondents are interviewed The visit 
may include checking on the interviewers' problem, progress, and accuracy of work. 

Accuracy of work is accomplished through spot checking. This may involve a visit ' 
by the evaluator to a few randomly selected respondents who have been interviewed 
They are asked if ti .ey were in fact interviewed, including apprbximate length of inter- 
view time. However, when resources do not allow you to provide_3ctual field supervision 
you may either use the telephone or postcards as means of follow-up. Select a random ' 
sample of respondents and contact them by telephone or by mail using an easy-to-fill- 
out, self-addressed, stamped postcard verifying whether they participated in the interview 
These procedures are important in checking whether the interviewers are in fact doing 
their, job and not fabricating results. 

5. Reviewing completed instruments by tfie evaluator. It is suggested that after the inter- 
viewers have turned in their completed interviews (at different points of time during 
the interview period), the evaluator go through each completed interview meticulously 
and check jf all necessary questions are answered. Check also to see if there are no 
missing pages. Staple each set securely. The front pages of those that passed the 
inspection shov.i;J be initialed and dated. Those incomplete and/or highly doubtful 
questionnaires must be completed and discarded, respectively , and responden ts should 
be mterviewed again whenever possible. After the questionnaires are insepcted (i.e., 
those with complete stivers and pages), they should be deposited in one place. If ' 
Dossible, tie all returns from one place and label them appropriately. . 

Checklist 12 is designed to help you deal with the problems of minimizing interviewer's bias. 

Dealing with Social Desirability Bias 

Another important source of data diitorilon is desirability bias, which is "a tendency to offer 
socially desirable answers ... to answer qurstions in a way that conforms to dominant beliefs or 
patterns among groups to which the respondent feels some identification or allegiance" (Dillman 
1978, p. 62). For example, some of ypur respondents may be located in communities where particular 
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CHECKLIST 12 
Minimizing Interviewer's Bias 



Instructions: Please respond appropriately to the following questions: 



Yes No 

□ □ 



1. Is the interview standardized allowing the interviewer as little free choice 
as possible? 



2. Does the intervievver's training include the following aspects of the study? 
n • Objectives of the evaluation 

□ □ • Objectives of ever y questi on 

□ □ • Sources of interview bias ' 

3. Is the interviewer trained in the following areas? 

□ □ • Making initial contact 

n □ • Securing the interview \ 

□ □ • Using the questionnaire \ 

D □ • Probing and other interview techniques 

n □ • Recording the interview 

□ □ • Editing the interview 

□ □ 4. Is there a mechanism for providing a regular check of the data collected? 
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schools are known to be poUlarj Socially desirable response bias is manifest when former vocational 
education students say the^ are satisfied with their training, even though they are not because 
they do not want to express an dpinion that runs counter to the dominant view. 

There is a greater prohabilify of social desirability bias in face-to-face interviews than with use 
of a maMed questionnaire, ^ucha probability can be compounded if the interviewer is personally ' 
known by the respondents or if/the interviewer is known to be affiliated with the school The 
following techniques are suggested to minimize social desirability bias: 

■' I / ■ i • 

1. Use of telephone (interview or mailed questionnaire 

/ I / 

2. Emoloyment of interviewers not personally known by the respondents. The use of 
vocational^teajchers or guidance counselors as interviewers, even with the use of the 
telephone, is discouraged. 

. How/to Increase Generalizability of Interview Dat^ 
/ |/ • ■ ■ ' / . 

Good methodology d^i^fctates that you should strive to get HOO perceht usable returns (i e 
complete, accurate answers) from every selected respondent, especially if you have a random ' 
sample. In cases where only a certain percentage of the sample repNy or /in' cases where some completed 
interview schedules,have be discarded because the answers are inaccurate, it becomes inappropriate 
to generalize the total population from the data collected. \;' 



/ , 



In this section, discussion focuses on strategies for increasing the rate of participation to mcrease 
generalizability of interview data. The following procedures are offer;bd ^or the evaluator's considera- 
tion: . . I I I ' \ 

1. Identify sufficient resources from which to obtain updated addresses of former 
vocational education students. // V 

2. petermine the best place to send precontact letters and the best site for the interview. 

/ ■ ' ' \ " 

3. Design'all correspondence in the most professional and appealing format 
, / / . . / ''^ \ 

4. Considei< using the telephone as a method of precohtact. Inform the respondents of 
' the pbrpose and sponsor of the study and explain that a trained interviewer will call 

at their addresses. / \ 



Ensure that interviewers are trained to make proper initial contact-both at the door 
and inside the house. The Institute for Social Research at the University of IVl)chigan 
(1976, p. 7) offers the following advice: / A 

/ / . r ■ \ ' \ 

/ At the doorstep you should state the course of action which you 
/ /desire rather than ask permission for the intervievy. For instance, 
/ / instead of asking, "May I come in?" to which a respondent could 
/ / easily reply "No," say, "I would like to come in and talk with you 

about this." Avoid questionr> such as "Are you busy now?" or 

"Could I take this interview now?" or /'Should I come back?" 

Questions which permit undesired responses can lead or even push 

a respondent into refusing to be interviewed. \ 



\ 

■• \ 



\ 



Summary 

The use of the interview for collecting data to measure training and job satisfaction provides 
two distinct advantages over the mailed questionnaire: it permits the study of complex and 
sensitive topics, and it allows the interviewer to check on the accuracy of answers (figure 6). 
However, there are also serious disadvantages: it is subject to interviewer bias, and it is costly. 
Thus, the evaluator needs to consider several factors before deciding to use the interview. Like 
other data collection methods, the evaluator needs to be thorough and exacting in the process to 
ensure a high degree of confidence in and generalizability of data collected (figure 7 and figure 8). 
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FIGURES 

interview; Advantages and Disadvantages 



intet'vjew: 

Data an/ collected by 
talking with people 
either face-to-face or 
by teiephone. 



Advantages 

> Does not require reading or 
wrcting by respondent 

Is adaptable to unforesieen 
circumstances 

^ Provides opportunity to 
obtain a representative 
sample of respondents 

Permits study of complex 
and sensitive topics 

Permits int^rrviewe* to 
check on respondents 

Permits the responding 
environment to be 
structured to some degree 





Should i usf; it? 



Considerations^ 



Disadvantages^ 

Requires trained int^^ Viewers 

Is subject to unpredictable 
interviewer bias 

Tends to be difficult to I 
structure because of large / 
response volume^ ' 

High cost involved 
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\ FIGURE? 
Increasing Confidsnce in intervSew Data 



Goal: Increasing reliability 
of data collected 
by interview 




, Increasinsi confidence m the^ 
.utility of data collected for 
decision making. 



Decrease measurenient i 
irror by 




i 



k 1 



Minimlzingjpterviewen's bias through 
1 properselectipn and training of 
interviewers, standardik:atipn of 
intervievi?, proper use of probing 
techniques, proper use of schedule 



Ensuring question construction: 
content, st octure, wording, and 
response options 



[ Minimizing scaial desirability bias 
i by employing mterviawers not 
, personally known to the respondents 
and by use of the telephone 



3 — '^s. '"^ "X ^ ano^y^^se OT tne teiepnons 
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FIGURES 

Increasing Genernlizability of Interview Data 



Goal: Increasing the 

generalizability of] 
data collected 
by interview.. 



Increasing utility of 
evaluation data for 
decision making^>^ 



Increase external validity^.^ 
of data by 




Sparing no effort to update 
addresses of respondents 



Using precontact techniques 
such as sending precontact 
letters and precontacting 
by telephone 

T ( 



T 



\ Employing proper techniques ^ 
in making initial contacts 
and securing the interview 
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CHAPTER VI 
SOME QUALITATIVE METHODOLOGIES 



Chapters 3 and 4 presented the tv^j most common methods of data collection in survey ' 
research-mailed questionnaires and interviews. These methods make it possible to quantify and 
generalize results and to present evaluation findings to information users. However, the under- 

qiialitative methods by evaluators has been the subject of increasing criticism (Wolf 
et al. 1979, p i). One of the reasons is that results tend to be oversimplified, and therefire some- 
times are misleading. A presentation of data resulting from these types of studies lacks detail which 
would provide the reader with a holistic view of a program, r, 

In many cases, it becomes necessary to use qualitative methods as a complement or supple ient 
to quantitative measures." Complex situations are better understood when dealt with in terms of the 
dynamic? of the social processes involved. Use of qualitative as well as quantitative measures provides 
a better understanding of the multiplicity of causes associated with given outcomes in vocational 
education (McCaslin 1978, pp. 4-6), and of the fact that there are "multiple realities and multiple 
perceptions and interpretations" (Wolf et al. 1979, p. 3). Bogdan and Taylor (1975, p. 4) say: 

Qualitative methodologies refer to research procedures which produce descriptive 
data: people's own written or spokert wcrdi and observable behavior.' This 

apprc/ach ... directs itse lf at settings and the individuals within those settings 
hohstically; that is, the subject of the study, be it an organization or an 
individual, is not reduced to an isolated variable or to a hypothesis, but is 
viewed instead as part of a whole. 

It is the purpci-e of this chapter to introduce the reader to participant observation and 
unobtrusive measures-qualitative nethodo'ogies-that may be appropriate in measuring training 
and job satisfaction of former vocational students. Unobtrusive techniques are those data collection 
techniques that do not require contact with the individual-observation, physical trace measures 
and research of archives or records (Kester 1979, p, 53), Discussion in this chapter is focused on'the 
following topics: types of unobtrusive measures that may be applicable in measuring training and 
job satisfaction of former vocational students, advantages and disadvantages of using qualitative 
data in measuring training ana job satisfaction of former vocational students, and tachniques 
employed in data collection by observation. (For further discussion on qualitative iviethods refer 
to Kester 1979 and Spirer 1980.) 

.• ■ ^ ■ - 

Participant Observation 

The use of participant observation techniques is relatively new in vocational education evaluation. 
A review of 1 ,500 studies on the effectS/6f vocational education (Merteils et a!., 1 980) showed that 
qualitative mett^ods for measuring training satisfaction and job satisfaction are virtually never used. 



\ 
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Participant observation has been traditionally an anthropologist's methodology It is used 
synonymously with field observation, qualitative observation, ethnographic techniques and direct 
observation-all of these referring to a condition wherein the researcher is immersed in the system 
under study vvhile in the process of data collection. Three advantages are cited by Bouchard (1976, 
p. 385): ' 

. . . [First] , it focuses the researcher's attention on the behavior of individuals 
rather than simply on their verbal interview or test-taking behavior. ... A 
second advantage is that it tends to force the researcher to look at the whole 
[person] , the whole organization, and the whole environment [social and 
physical] in an integrated way . . . [and third] , it puts the . . . [evaluator] in 
the context of discovery. 

The evaluator is, however, warn'ed that "the price of doing fieldwork is extremely high not in 
dollars ... but in physical and mental effort" (Bogdan and Taylor 1975, p. vi). It is often necessary 
for the evaluator to lead two lives simultaneously-"as a participant in whatever little world is under 
study while, at the same time, attempting to make sense out of the world as an observer" (Bogdan 
and Taylor 1975, p. vi). The succeeding two sections detail two most common methods ( Lof land 
. 1971) of the participant observation technique. 

Unknown Observer 

in this situation the observers do not idenrify themselves. They, therefore, assume incognito 
roles and remain unknown as such to many, if not all, members of the v.,:ting (Lof land 1971 p 94) 
For example, a group of Hispanics who arte^former vocational education students may be employed 
by a firm. Hispanic observers may be hi. ci ib^ather data as unknown observers.\They may take 
jobs, and so join the group foi^the purpose of collecting data. In this type of situation, to gather data 
on- job satisfaction as unknown observers may be the most effective method. 

Advantages: 

1. Richer materials provided. As one of tha group, the observer becomes friends wi th 
soma members and therefore is quite likely to be entrusted with their intimate 
thoughts and feelings, a/situation that is unlikely if the observer is known. 

■. /' ■ ■ 

2. Possibly the only way to gather information, Certainjettings may not be amenable 
to other data-gathering methods. In.this case, "being an unknown observer may seem 
better than doing nothing" (Lof land 1971, p. 94). 

Disadvantages: 

1 . Ethical questions. There have been some objec- ons as to the morality of observing ' 
and analyzing pecple^vyithout telling them this is taking place. However, some 
sociologists feel that "as long as reports conceal names, locations, and the like, 
thereby preventing the analysis from being used against the particioants, there is no 
harm done" (Lofland^971, p. 94). \ 

2. Limitations on observation. The observers play 'specified roles so they may be cut off 
from valuable channels of communication of information. Additionally, since a good 
part of the observers' time is spent performing a role, they are therefore constrained . 

in terms of time and place from doing observational work, including jotting down notes. 
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3. Biased viewpoint effect. It is possible that the observers may "selectively expose 
[themselves] to the data or selectively perceive them and, worst yet, shift over 
time the calibration of [their] observation measures" (Webb et al. 1966, p. 114). 
Further, it may be difficult for the observer to disengage or disentangle their 
emotional involvement. 



\ 

Known Observer 

Formal or informal permission is often needed from "gatekeepers" to allow observers to visit 
the employees in their working environment ior the purpose of collecting data. Both the employer 
and the enriployees (former vocational education students) know that observers will be visiting them 
to collect information. There are both advantages and disadvantages ?>f this method. 

Advantages: 

1 . Greater degree of freedom to collect information. Since the observers do not have an 
"extant role in the setting, there is a greater degree of freedom for [them] to move . ' 
observe, and ask questions" ( Lof land 1 971 , p. 95) . The observers can freely take Tt)'tes 
and schedule time. 

2. Fewer ethical problems. Since the subjects know that they are being observed, no 
deception is involved and few questions on ethics arise. , 

i 

Disadvantages: 

1 . Superficiality or marginality of observation. Unlike the case of unknown obi^ervers, 
who become the members of the group under study, "it can become quite evident to 
the known observers that, although they are in that world, they are not trufy a part 
of it (Lofland 1971, p. 97). 

2. Personal involvement bias. The observers who become personally involved with the 
problems of employers or employees are likely to be less objective, and like the 
unknown observers, suffer from biased viewpoint effect. Writing an objective report 
and analysis becomes difficult. 

3. Observer effect The known observers can produce/hanges in behavior among the 
observed groups, thus affecting the validity of the^ata recorded. As a strategy, Webb 
et al. (1966, p. 113) suggestthat thf? obsiirver contaminant be permitted to wear off 
and that analysis of data start st ^:;r->Ment to the time when the effect is negligible. 

4. Problem of acceptance. Related to v^e problem of observer effect is the problem of 
acceptance. The observers may be viewed as intruders, or worse yet, as tools of 
management sent to spy on the employees. If chis happens, it becomes difficult for 
observers to get honest reactions. 



Strategies for Successful Obs6 ation y 

/ 

Successful observation requires a good observer-a person with the n^-c3ssary personal and 
technical skills. Thorough planning and preparation are also important. Tf.f person doing the obser- 
vational work must match the general and the specific requirements of irj.> i^r. cial settings where 



observation is to take place, which is sometimes difficult due to the diversity of personalities among 
those being observed. However, under no circumstances must observers with personality styles that 
make it difficult for them to "blend" with the observed group be hired. 

In addition to certain personality requirements, a good observer needs certain special skills for 
successful observation. These include an ability to listen, to make notes (both mental and written 
on and off the field), and to write reports (see table 4). Observers should know what to look for 
Thus, evaluators hiring inexperienced observers need to pay particular attention to their training. 
There should be adequate practice before inexperienced observers are assigned to actual field work. 

We are now ready to move into the next section of this chapter. However, befo- e you begin to 
read the next sectiorixtake a few minutes cf your time to examine your strategies for successful 
observation by answermci the questions in checklist 1 3. 

Unobtrusive Measures ^ 

As defined earlier, Unobtrusive measures are those data collection procedures that do not require 
contact with the respondents, thereby avoiding contamination by reactivity, an integral part of inter- 
views and participant observation. However, "lack of reactivity in a method^does not mean that the 
data generated are valid" (Bouchard 1976, p. 399). Four classes of unobtrusive measures are physical 
trace, archives, measures gathered by hardware, and sifTiple observation (Webb et al., 1966) The 
first three measures are/discussed.in this section. 

Physical Trace 



^ Physical trace techniques are the examination of physical evidence that gives clues of stud( 'ts' 
training satisfaction and job satisfaction. "They are, therefore, most prone to misinterpretation and 
should be used with caution" (Bouchard 1976, p. 399). Acting like a detective, the evaluator is 
trying to identify for physical evidence that lead to conclusions. A few of the possible indicators of / 
students' satisfaction with their training include- / 

/ 

• wear and tear on instructional books and equipment, ' 

• number of books checked out, / 

• amount of consumable instructional materials used. 

Archives and Records 

Documents and records c^n be very useful as sources of data to determine students' satisfaction 
w'th their training and job. However, the evaluator should exercise "prudence and caution because 
/ these materials can never be taken at face value" (Bouchard 1 976, p. 400). For example, a record of 
y leave of absence may not indicate the reason leave was taken. A variety of causes, such as health 
problems, are possible. Some examples of documents and records to investigate are as follows: 

Satisfaction with Training Job Satisfaction 

, ■ • Letters from former o Daily time record 

vocational students a Leave of absence records 

• Attendance records at ' / • Requests for transfer or promotion 

alumni meetings and reunions • Pay levels 

• Turnover rate 

• Record of strikes and :^ ievances 
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TABLE 4 

Some Rules of Thumb in Participant Observation 



Entering * In the initial stage of the fieldwork, it is not advisable to challenge the 
the Field behavior or statements of subjects or to ask questions that are likely to 
put them on the defensive. 



• Observers should conduct themselves in such a way that events occurring 
during their observations do not significantly differ from those occurring in 
their absence» 



Establishing 
Rapport 



Developing 
Relationships 



• Probably the easiest way for observers to gain rapport with their subjects 
is to establish what they have in common with them. 

• To participate in activities of the subjects ?s helpfuL 

. ^, , 

® Researchers should refrain from developina close relationships with 
individual subjects while they do not have a good grasp of the nature of 
relationships in settings. 

• Where it is essential for them to establish rapport with a few selected 
subjects initially, they should be willing to withdraw from those relation- 
ships as circumstances demand. 



Asking * Questions should be asked in such a way as to enable the subjects to talk 
Questions about what is on their minds and what is of concern to them, without 
forcinig them to respond to the observer's interests, concerns, or precon- 
ceptions. 



Field notes should be recorded after each and every observation period, 



Field Notes 



as well as after more casual contacts with subjects outside the setting. 

• Observers should d.evelcp a level of concentration sufficient'to enable them 
to commit ^o memory everything they see, hear, smell, and think. 



Note: Ideas in making this table were taken from R. Bodgan and S. \ ,ylor. Introduction to 
QuaUtative Research Methods, New York: John Wiley & Sons, 1975. 
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CHECKLIST 13 
Strategies for Better Observation 



r 



Instructions: Take time to review your strategies for improving observation data gathered 
by answering the following questions. 



Yes No 

□ □ 
G □ 

□ □ 



Personality Considerations 

1. Is the observer reasonably able to get along with the subjects under study? 

2. Do the observers like the subjects, even though they may not agree with 
their views? 

3. Can the observers know how to become inconspicuous and inoffensive in 
the setting? 



Technical Sl< ills 

□ □ 5. Do the observers know how to take good mental notes? 

O □ 6. Do the observers know how to translate mental notes into written notes? 

□ n 7. Do the observers know the technique of taking good written notes? 

8. Were the observers trained in the following mechanics of making full field notes? 

□ Q • Writing promptly 

□ □ • Writing effectively 

' • Dictating, handwriting, or typing reports 



ERIC 



80 



8d 



Hardware 



Although the review of studies since 1968 on the effects ot partlcipc^ting in vocational education 
(Martens ^et csl. 1980) did not show ^he uso of hardware as a data-Tithering techrrique, recent^advances 
in dark photography and microrecorders make it possible to gather quality data and information ^ 
with th3 use of hardware. Some of the possibilities (Bouchard 1976, p. 401) include the following: 

1. Use of microtransmitters to record conversations among or between employers 
and employees 

2. Use of hidden tape cassettes to record conversations among or between employers 
and employees 

3. Use of photoelectric cells to record movements of people and some types of objects 
past key points 

4. Use of hidden camera or dark photography to get ^^ictures of employees during 
work period 

At this point, a word of caution is in order for those who wish to use hardware as data collection 
tools. There are a number of laws that protect the privacy of ii;idividuals. Violations are criminal acts 
and subject to lawsuits. Additionally, many researchers feel th^at collecting data from people without 
their knowledge is unethical. Thus, before deciding to use ha/dware as data collection tools, the issues 
of possible criminal violations and ethics need to be resolved. 



Summary 

The use of participant observation and unobtrusive measures are relatively nt;w in vocational 
education evaluation. !n fact, studies on training and job satisfaction of former vocational students 
during the last eleven years show that the employment of qualitative evaluation methodology is 
practically nonexistent. However, there is currently a growing trend for the use of qualitative data, 
in addition to quantitative data, to provide better understanding of programs being evaluated. 

The use of participant observation and unobtrusive measures requires the evaluator to 
recognize possible sources of internal invalidity and to apply appropriate controls. In addition, the 
evaluator must be able to recognize the limitations as well as the strengths of data gathered by 
qualitative means. 
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CHAPTER VII 
DATA ANALYSIS AND INTERPRETATION 



^^^e^y'i:^^^!:^!^^^ -^J^^^^^ of former vocationa, student, you are 
the data so they can be understood cllSed ald^.^r??'!'^'' °^ summarizing and arranging 
problem by condensing your lara^^Zumof^ar^^^^^^ ^" ^«'P ^olve this 

"numbers can be talked about, rfmerberL and^^^^^^^ """^^ers so these 

opinions, or developing theories'' Tf"^^^^^^^^^^^ f°-ing 

used t'otetr;;";^^^^^^^^^^ techniques that can be 

include descriptive statistics for summarizing data sv^^^^^^^^ r^"''"""' These 

generalizations and interpretations systematically and inferential statistics for making 

statis'cLrtoofdls^ctre^in^^^ ? in-depth treatment of the 

bibliography (I.e., Tatsuoka 1 971 ; E' i 976^^^ ''^'"'"^ '° ^^'^'^^^d s°"^ces in the 

Descriptive Statistics for Summarizing Data 

incluj^^rd^d^r tT^^^^^^^^^^^ Your data may therefore 

or^SsSS^^Si-^^^^^ 
and salary; and other variableTof int^^^^^^^^^ 

rnanzed by use of descriptive statistics for SnTesem nH ?n h° «V^!f"^^tically sum- 
presents some statistics designed to help you in thPs area ' ' ^^'^P'^^'^'^on. This section 

Frequency 

know'the^y~,^TlVr^^^^^ ^or example you may want to 

predictor variables like sex: designations ethnicitv a^^^^^ vocational students) in terms of some 

Besides the usual frequency talkie distSutTon^ h^^^^ programs in which they are enrolled, 

ways that will facilitate comnrlhpn r:..''::"!f ^^^L^" .^^ S^^Phi^ally presented in a variety of 
addition, frequency data can Te presen^^^^^^^^^^^ °^ a histogram, polygon, or pie graph. In . 

or cumulative frequencies ^our ?m o da the ent^a^thlr' ""'^^"^^^e frequencies, 

audience receiving the reoort are thP mlin? f J I !u ^""^ °^ ^^^^V' the kind of 
best way to present you rXa Figure 9 nH^^^^^^^^^ ^ '° conslciere6 in determining the 

illustrate percentage dltribution.^^ ' °^ ^''^^'^^ presentations to 
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FIGURES 

Graphic Presentations of Percentage Distributions 
(Decree of Satisfaction with Salary of Former Vocational Students) 



Histogram 

50% 

40 

30 

20 

10 



Polygon 
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2 3 4 

Level of Satisfaction with Sialary 



2 3 4 

Lovel of Satisfaction with Salary 




Pie Graph 



Legend: 



1. Highly dissatisfied 

2. Dissatisfied 

3. Neighter satisfied nor dissatisfied 

4. Satisfied 

5. Highly satisfied 
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Measures of Central Tendency 



\ 



Measures of centra! tendency, which means aver.yge, include the foNowing: arithmetic mean 
median, and mode Average is commonly defined as a number indicating the central value of a group 
of observations. The average offers two important functions: 

First, it is a shorthand description of a mass of quantitative data obtained 
from a sample. ... An average is, therefore, descriptive of a sample obtained 

at a particular time in a particular way. Second, it also describes indirectly 
but with some accuracy the populal;^n from which the sample was drawn 
(Guilford and Fruchter 1973, p. 42). 

Let us ts;-^ salai y satisfaction score as an example. The mean is the arithmetic average of the 
TnJi% °f the rt.pondents. Th^ mean is computed by adding all the scores of the different respondents 
and dividing it by the total number of respondents; thus, the mean can be heavily influenced by 
extremes, particularly ina snpall number of cases. The median is the midpoint between the hiohest 
and the lowest score, above pr below which are half the respondents. The mode is the score with the 
highest number of . espondents. The example that follows illustrates the three different methods of 
f:easuring central tendency. 

DATA COMPILED FROM A STUDY OF 
FORMER STUDENTS' JOB SATISFACTION 



Salary satisfacxion 
score (x) 

5 
4 
3 

2 
1- 





Frequency 


Code 


(f) 


Highly satisfied 


10 


Satisfied 


20 


Neither satisfied 


nor dissatisfied 


35 


Dissatisfied 


18 


Highly dissatisfied 


12 ... 


Total number of scores = 


95 



Product 
(flix) 

50 
80 

105 
36 
12 



Sum of all scores = 283 



Mode - The inost frequently occurring score = 3 . 

Median- The score that is midpoint between the highest and lowest score = 2 

Mean - Sum of all scores divided by the total number of scores = 283/95=2.98 

Where the'distribution is reaso.iably symmetrical, the arithmetic mean is usually preferred and 
IS most useful because it is generally the most reliable or. accurate. It has stability and consistency 
• thus. It IS better suited for arithmetical computations, as will be seen later in the chapter. On the ' 
other hand as suggested by Fitz-Gibbon and Morris (1978, p. 26), the median is used rather than 
the mean if (1 ) you will not need to perform statistical tests requiring the mean, (2) if you need a 
quick estimate of group performance, or (3) if the distribution, is not symmetrical. Additionally if 
the level of measurement of your variables is nominal, only the mode is meaningfui. 

Measures of Variability 

Measures of variability are aho known as measures of dispersion, heterogeneity scatter or 
spread. It is used to describe important characteristics of distributions, i.e., the variability of'the 
scores. Using the salary satisfactbn scores as an example, measures of variability answer the 
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arS^ThP^jri"'^ V'°''' contribute to the mean salary satisfaction score of the 

K«» J^V°*^' '^"^^ ? ^^''1^ detern-ined, but it is also a highly unreliable measure because it is 
based only on two values. Again using <-alary satisfaction score ao an example, the ranSi computed 
by subtracting the lowest reported score from the highest reported score. Different grouPs of 
respondentc and types of vocational programs can be only crudely compared on the basis of their 
range of salary satisfaction scores. on mc oasis or tneir 

thp nl''jr"''''''M commonly used indicator of degree of dispersion and is 

the most depenaable estimate of the variability in the total population from which the^mple 
canr.e (Guilford and Fruchter 1 973, p. 65). The standard deviation of the sa^rT^tisfaSl^n scores 

thVrJ ¥r'°"n 'T""'' ' T'''''' '""'^^^'"9 "^^'^ '^^ ^'^^^^ are spread out' round 
Jevirticn'J. a^o.'n'V'' '"^ ^P^^^' Knowing the standard 

aeviaticP u5 a group of measurements performs two functio.ns (Morris and Fitz-Gibbon 1978, p. 30): 



* ! .; provides a good means for describing the spread of certain measures 
.e g., income or perception on the quality of training) obtained from 
the administration of a particular instrument. 

o h provides a basis for later statistical procedures that you may want *o 
perform, such as a test of the significance of differencet; between qroup 
means. 

The example be!ow wil! help you differentiate between the two types of measures: 

Sa/ary satisfaction Frequency 
score (x) (x^) (f) 

5 25 10 

4 16 20 

3 9 35 



4 18 



(f)(x) 


(f)(x^) 


50 


- 250 


80 


320 . 


105 


315 


36 


72 


12 


12 


283 


Sum = 969 



1 1 u 

" = 95 Sun! ■■ 

Raoqe = highest sccre - lowest score 
=5-1=4 

Standard deviauon -■ n(fx^) - (fx)^ / n 
- 100 (969) - (283)2 
= 1.36 



Some, tnfurentlal Statistics for Making Generalizations 

Evaluators and users of evaluation information are keenly interested in the generalizations and 
conclusions arrived at on the basis of the data collected from the sample respondents. iVIore than any 
facet of the evaluation report, perhaps, the generalizations and conclusions have the greatest influence 
on decisions affecting policy and program improvement. Thus, no efforc should be spared in your 
methodology ttf enable you to achieve defensible and clear generalizations about your evaluation 
study Inferential statistics is the "body ot methods for arriving at conclusions extendinq beyond the 
immediate data" (Hays 1973, p. v). It aids in drawing conclusions about a population from a sample 
or group ot samples. Recent advances in computer programs have made it possible for evaluators to 

/ 
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answer difficult questions which would not have been possible a decade ago because of the magnitude 
of mathemsticai computations. This section details some inferential statistics that you may use in 
arriving at certain conclusions in measuring the training satisfaction and job satisfaction of former 
vocational students. 



Univariate Procedures 



Univariate procedures are statistical techniques dealing witlp one dependent variable, regardless 
of the number of independent variables {Anderson et al. 1975, A 251). If you treat training satisfaction 
and job satisfaction as global variables, as in the example that follows, then your inferential statistics 
deal with univariate proced;ires such as the t-test and analysis of Uriance. Multiple regression analysis, 
although classified by some vialisticiansas a multivariate proceduVe (Tatsuoka 1971, p. 1), is included 
in this section to differentiate it from other statistical analv^es deLling with multiple dependent variables. 

Question: To what degree are you satisfied with your^j^/ocational training 
^ as having prepared you for your first job? (Circle one) 

5 Highly satisfied \ . 

4 ' Satisfied \ 

3 rjeither satisfied nor dissatisfied \ 

2 Dissatisfied 

1 Highly dissatisfied 



You should be aware that in using the foregoing statistical tools,\certain assumptions need to be 
;met and that if certain'as?umptions are violated, there are adverse conpquences. Additionally, you 
should be aware of the specific functions of a specific statistical tool. A good evaluator should then 
be able to answer the "what," the "when," and xh^ "how" of each statistical procedure employed in 
the data analysis. Table 5 details some univariate' procedures, their uses! and ttieir limitations. 



Multivariate Procedures 

Multivariate procedures refer.to a series of statistical techniques forlanalyzing a set of dependent 
or outcome variables. Earlier discussion in this handbook shows that tr/aining satisfaction or job 
satisfaction have multiple objectives, aspects, or components. Thus, "job satisfaction can be seen Jn 
terms of an individuaTi; satisfaction with his pay, his supervisors, working Conditions, and the 
amount of variety his job provides"^ (Weiss 1976, p. 327). Likewise, training satisfaction can be 
viewed ?s a multivariate measure of former students' satisfaction with trairting facilities, curricular 
contents, teaching methods, or cchoo; sen'ices as shown on the^next page.' | 

Multiple variable investigation provides good payoffs in terms of provilding the evaluators and 
u^sers of evaluation information with co;isiderably more information than that provided by univariate 
procedures. However, "ii doas raise problsm5 which dealing with a single valriable does not'' (Weiss 
1976, p. 328), because it becomec difficult to con^prehend and synthesize ttle mass of all the available 
information. For example, the question of who are the former vocational students who are the "most 
job satisfied," or for that matter, the "most satisfied with their training," is ioo complex for an 
observer to answer. One former vocational student might be hi^jhly satisfied With the training facilities 
and curricuiar contents, but highiy dissatisfied with teaching mv-thods and scpool services; another 
respondent might have a reverse rating. 
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TABLE 5 
Some Univariate Procedures 



Statistical 
Techmques 


1 

Functions 


Assumptions and 
Limitations 


i. t-test 


Used mainly to test hypotheses 
about xhs difference between 
two means. It can ans.ver such 
questions as 

• Does the training satisf':;CCion 
of postsecondary students 
differ from that of secondary 
students? 


• Random sample of 
respondents from 

. two normal popula- 
tions 
» Equal Ns 

• Equal variance 

• Dependent variable 
:n°asure isMntervc' 

if or ratio 

;/ ' ■ 


2. Analysis 
of 

Variance 
(ANOVA) 

J / 


Used to test hypotheses about 
the differences between three or ,/ 
more means. It can answer such 
questions as /' 

© Are the different groups of/ 
former vocational students 
significantly different in^ 
their job satisfaction?/ / 


X • Random <;amnlp nf 

respondents frorr^ three 
or more normal popu- 
lations 
« Equal variance 
• Dependent variable 
meai jre is interval 
or ratio 

i ■' /■ 


.3. Linear 
Multiple 
Regression 
Analysis 
(LMRA) 

1 

/ 

■ / 


■ // ' 

Two or more criterion variables 
are used to predict a univariate 
depencent variable. It can 
answer such questions as 

• W at situational variables 
(e.a, salary, type of job, high 
school GPA, etc.) are asso- 
ciated with the training 
satisfaction of forrrier 
vocational students? 


• Random sample of 
respondents 

« Large sample (e.g., 
60 or more); 

• Normal distribution 

• Appropriate for data 
that do not;violate the 
assumption of Mnearity 

J * It is a maximization 
! procedure,; It tends to 
capitalize on sample- 
specific covariation to 
give results that are 
artificially inflated due 
to unique characterif- 
tics of the sample.. 
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In this regard, multivariate procedures can help the evaluator by providing an organizinn 
function to permit better understanding of the data. In addition, it can help the evaluator in" 
making better predications (Weiss 1976, p. 328). Tatusuoka (1971, p. 1) says: 

Multivariate analysis is concerned with a group (or several groups) of individuals 
each of whom possess values or scores on two or more variables such as tests 
or other variables. We are interested in studying the interrelations among ' 
these variables concerning the populations from which the sample qroups were 
chosen. ; » 

■ \ 

Four multivariate procedures are explained in table 6: discriminant analysis, factor analysis 
canonical correlation, and multivariate analysis of variance (see figure 10). You are encouraged to 
examine these.statistical tools in more depth before attempting to use them. Further, since multi^ 
variate analytical procedures require the use of computers and special computer programs you 
may also like to consult the personnel handling the computer needs of your agency for advice and 



Question: Rate the degree to which you are satisfied with each of the following 
aspects of your vocational program in preparing you for your first job. 



1. Training 
Facilities 

2. Curricular 
Contents 

3. Teaching 
Methods 

4. School ' 
Services 



Highly 
Satisfied 



□ 
□ 
□ 



Satisfied 

• n 
n 
n 
n 



Neittier 
Satis fiecf 
nor ' 
Dissatisfied 



n 
n 
n 

D 



Dissatisfied 

□ 

□ 



Highly 
Dissatisfied 



□ 

n 
n 
□ 
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TABLE 6 



Some Multivariate Procedures 



Svtatisticat 
Teohniques 


Use 


Assumptions and 
Limitations 


1. Discriminant 
Analysis 


To find the linear combinations of 
variables which best dominate among 
a set of predefined groups. In measuring 
the training and job satisfaction of 
former vocational students to:; 

1. Determine which ways the students 
in the different vocational programs 
are most dissimilar on a given set of 
variables like training satisfaction, 
job satisfaction, etc. 

2. Predict job termination from the 
multivariate measures of job 
satisfaction. 


1. Appropriate only for data that 
do not violate the assump::on 
of linearity. 

2. Limited by random samples i 
of respondents. 

3. As a maximization procedjre, 
it is highly susceptible to 
sample-specific covariaticr,. 


2. Factor 
Analysis 


1. To summarize data and to identify 
structures in the data not immedi- 
ately evident. The summarization is ' 
a much more ambitious undertaking 
than that involved in a simple 
statistical summary. 

2. To summarize and clarify all the 
interrelationships among the 
variables involved in a study. 

To avoid factor-analyzing worthless data, 
lesearcners snouid tirst test the«r inter- 
correlation matrix for statistical signifi- 
cance and proceed to perform factor 
analysis only if they conclude that their 
data represent significant deviations 
from a random intercorrelation matrix. 


1 . Random sample of 
respondents 

\^2. Interval type of data ■ 

3. Multivariate normality 

1 



/ 
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Table 6 (contrnuad) 





Statistic-^i 
Techniques 


v- 


Assumptions and 
Limitations 


3. Canonicc 
:orre!3ti^^" 


Canonical corre ation is the extens ^ cf 
linear nnultipie regression to the ca i r 
multiple criterrrjn variabies. It is ap irc- 
priate for the i:iuation in which a s yr 
continuous predictor ^THriables is tc ^ 
related to a set cri™ion variable 
-ssentialiy, a sen^rats sei of v^/aigh' 
ound for each of variHbls:^ such - - 
^ these weights are lised rind thr 
veighted sums eacn het:. :ne tv\/c 
weighted sumsv/.f{li hsrcaiTE r^d a 
ighly as possbie. Per axarrv^i:. cc r:! 
orrelation car Dt- in r-^r^ir^'v 
multivariate azzai..^\ ..icasure 
:tisr3cticn WHirh r^ny, vvonc-inc! rcna: ■ 

*rom a se-'or ^airrng sr 
■• i::iires or GiT-^-'-raph- ::r;r.tr 


^ / 

1. it is appropriate only tot data 
that do not violate the 
assumption of linearity;. 1.^:., 
all realtionshipis betwesnrsch 
criterion variable, wittT!Tnn:e 
predictor sat arrd w*nthr-"~-e 
criterion set =re linsar. 

2. Random sampriias of 
respondents 

3. fr: is a maximization pn-najre; 
^ tnus, it tendsxD capftafiZE-: on 

sample-speciffc covarraTrnn to 
give results that are ^, . " dally 
' inflated due to uniq/i^E larac- 
ceristics of the samiple. 


i 

I 
i 


- 'OVA is tr.;: -':;nivar:-3r£ ■ 
B analysis - -ariancr i A : " ^ 
■^r. riate star.::; •::3 and ir-TrcEr: ■ 
use - testin ,"-:^^differe -es HiMicj'-^ — 
me£i:jjTaj on Tcs^enden: rouDs 
iecn . vfANO^^-^plies tc a:sEz or -i- 
vvhen^aon-^L- Al\!0\ test3i'?r 
diffErejTcss a zrr'-the means cr, sa :v-: 
grou-rnf fc ~- ocationsi studer :5, a 
one- MA can test the differe::::^- 
^"Hiprg sets c ::-:s of several differed 
variani^s. Fc -pie, you might cor 
;3are tha difr- ?r:-^ on the muitivsria^: 
measures of ---.sE^iisfaction for differ— 
tvpes o^traiiTmn : rograms. MANOVA 
g:ve on£:ove-.v i mnificance for the group 
cifferences. 


1. Random sarr r 
respondents 

2. Normal distnir on 

3. Equal variancf 

4. Dependent var mz£:::L~ 
is interval or "z 



:^T^^°°'l°fl''^=t^^dOrgam^ (Chicago: Rand McNally 

Inw D — -^^derson etal., Encyclopedia of Educational Evaluation 



College Publics: 

(San Francis=s:_:-^ey-Bass Pubiishsr, rsnj 

91 



ERIC 



90 



, FIGURH10 
Vision Tree for Selecting a Statistical Test of Bmmxi 



Is satisc: ron 

multivarc:] 
3r univsnzB 



Multivariate 




MANOVAof 




mm \ 


equal n's 


•„../ 


sieq^al n's ^ 



nunivann? 
ir miivanr- 



Mann-Whitney 
U test instead of, 
or in addition to, 
t-test 




^ t-test 
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Summary 



Two types of statistics are generally employed to anafvze snc inter 
statistics for summarizing data and inferential statistics for makinc gam 
tions. Descriptive statistics may include frequency counts, rnsasures r 
fT^dian, and mode), and measures of variability (range and nmandard at 

Where data are collec3d from a sample, inferentiai! 5rar:stics faci 
imimediate data. Whether uorrvariate or multivariate procedu-3s are err 
trstment of dependent v£Erfafale(s). Univariate proceduressirH statistic 
on= dependent variable, ar^ it Is appropriate vwrhen traininoi-iEtisfectic 
treated as global variables. ;hese include t-test. analysis of vc-iance, a: 
Multivariate procedures refer to a series of staiiistical technimias for ar 
variables. These include MANOVA for testing differences amr::ng se^s c 
factor analysis, and canontial correlation. The use of specif ic-tatisrics 
data and the objectives of irhe study. 



^ i-j cijsscriptive 
u . ifTtT^d interpreta- 
m™ncy (mean, 



or .:.i.3ions beyond the 
: I :jS32r::n..upon trie 
'cmffHiisaling with 
icv: !^-!3S3Ct^on are 
■tipa: -^rsssron analysis. 

a ^Hr:~ outcome 
^rv; :u™"nnir5ant analysis, 
c^nini::: - the type of 
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- CHAFTE 111 
DATA PRESENTATiOfv ATID U^?. 2ATI0IM 



-i^ on student satisfacticsi with their tr^inz anc -„ -"=rally are categorized as follow-up 
-rr ^wever, minimal attentian appears to giuen tcth- r interpretation and presentation. One 
ca -rrmiEs that this minimal ansntion contrihutss to tre- ok of utilization for accountability 

^rcn iraking, and program iriiiprovement. 

"niEshapter describes selected reporting sndi:^3£!v tech^qaes and strategies for increasing ■ 
■ii.2ation of etfaluation resnits. Emphasis is given tc? :ne prssentation and utilization of data 

iTormation on student satisfaction with their tranhiQ: anc job. Moreover, the focus is on the 
- onship of these data and information to accounssMty, cHcision making, and program improve- 
~ at the state and local education levels. 

However, this chapter prov^es only a basic intrcrtijuction -q the presentation methods and to 
r : -neory and practic^f evaluation data and inforrf-^tion utilization. The reader is encouraged to 
j3- r w Guide/ines and Practices far Improving the Utilization of Evaluation Results (Franchak 1981) 
: Handbook provides an in-depth treatment of th-; important and complex subject. 



Data and Information Presentation 

At the beginning of this section, we wish tomake a distinction between data and information 
fvr, rear and Koester (1 978, p. 85) offer such a distinction - 

I nformation comes from data, which sf3 logical representations of measurements, 
observations, and computations. Logics is here defined as orderly, intelligible, 
objective, and capable of forming accunsne relationships based on principles and 
rules of reason [italics added] . 

Furtner, they add that not all data are informatibr. that is, capable of being assigned a usefui meanir 

As discussed in the preceding chapter, the assessment of the training and job satisfaction of 
former vocational education students is a complex undertaking, and the process of data and infor- 
mation interpretation is equally difficult. This corai.'-r.s of taking the information resulting from the 
data analysis process and subjecting it to expert saru ny. Such an examination should result in an 
explanation of the displayed information in terms tr-rt are comprehensible to decision makers and 
other information users for accountability, policyrra^- ng, and program improvement. 

The interpretation and presentation of analyzec:asta on student satisfaction with their training 
and job can best be performed by the professional seht the evaluator, or data analyst. However, 
top managers in the organization at either the state cr ocal levels must feel comfortable with the 
fact that these individuals know vocational programs, are crediblis and competent evaluators, are 
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familiar w- : -sroblems facing the mnnagnrs, and have a deer 
Equally - : -n-tant to this process is :;he need for the svaiuatjr 
as integr?. iements of a comprehensive managemsnt inforrrr- 



:ncern fcr.the target audiences. 

cssta analyst to visv. / these data 
::'?n £-t/stem. • 



; ningand job 

reach: (1) the 
=3rget audiences 



Morr er, these data need tc be considered as only one -m=nt of informatic-' -ontained ir 
fn^ernS" '"h""^'''°" exan^pie, a pre .-~^y step in organi^^^^^^^^^^^^ 

■ .^^"J P^^^^«";^t.on ~u^crron of an MIS is decidin:: .rnch:program area Le., agriculture, 
technica, isalth), by functio- i.e., personnel developmerr- —cher, inservice o--urriculum 
developrr^n- : facilitates the nee: : of a specific education age^ - at the sSe (siA " bcal (LEA) 

hw 1 Jl'f °^ ^^^^ ^""^ information on student sa-::---ion wiith their t 

Dy LbAs aid - : may be m a --sriety of forms depending uin::-t!he usar they wi^ 

general pabiuc, education planners, (3) educational admini-nnDrs, and (4) oth"- 

such as the 302 :t education, program advisory committees, HTtn::legislature. 

Each;:-- - 
detailed renor 
students' stis 
selected aiidier: 
and recorrrme— 
to enable rher- 
purposes. 

Concaiv- 
former vocs- 
being prepar 

1. 

2. 



>£Sf audiences has different needs. The general puniic does not usuaii require a 
A one- to three-page report highlighting the resixrsof the assessmenr^-jf the former " 
XT. on with their training and job may be sufficie- Educational adrrmistrators and 
:ces may be interested only in a one-page executiv-summary-arid a li-of conclusions 
-ations. Educational planners, by contrast, may need a detailed "teci^nical" report 
- -0 -ecommend or develop specific strategies for urogram changes anc for other 

then, there are varying types of reports for presenting the results of assessing 
education students' satisfaction with their training and iobs-each type of report 
:ccording to the unique requirement of a specific target audience: 
jhlight report for the general public 

scutive summary for educational administrators of the advisory council 
-d board of education ^ 

5tailed "technical" report for educational planners, classroom/laboratory 
Tstructors, counselors/placement personnel, and other interested persons 



Reporting Recommendations 

their I^afnSrJnri inh °" the assessment of former vocational students' satisfaction with 

their traimng and job, a common mistake is to distribute the same report to all audiences As 
discussed in the previous section, one must identify the audiences whom the report is intended to 
serve and the purpose of the report-accountability, decision making, or program mprovement It 
^not uncommon for an LEA or an SEA to distribute hundreds of final report co^Ss with fifW i 
more pages. This practice is not only costly, but also of questionable value The general pub c 
rsfof rnformrn^^^ information and is probably confused by techhi^l data and a sheer 

mass of information. The administrator, on the other hand, who simply does not have the time nor 
technical expertise to review the entire^pontents to recommend policy action or determine Zgram 
lunTplnr'/T' ^."'^ educational planner, classroom/laboratory instructor ' 

counselor/placement personnel may need to know all the detailed information Thus a related 
consideration in the strategies for presentation and utilization must be the fength of the report 

a" I^Satand Spa"7l9^^^^^ '""^^r^" "^^^^^^' ^-^^"' 
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Top officials and management star were Tnore likely to ssk for short reports- 
program and project specialists wera more likeJy to requst medium or lonq ' 
reports in their areas of specializairnn. 

Further they conclude that differentiated reuonsibilitles require differing lengths of reports 
despite the preference for brevity. Proposed lengrh of reporS for decision makers are hort (one 
page}; medium (ten pages); and long (100 panss) (Brickell, AsJanian, snd Spak 1974 p S 

The alternative reports on student satisiEsimn with their trainrng and job satisfaction must be 
rjSt H'"if'r' °' r^''''' ^^^'''"3 At the state level. a!i thrie repor^^^ m°ght be 

TnZl i ''''' '"'1"''°''' the^ment of student ^isfaction with th^ ?tra^ning 

and job of a I former students or a sampra of rrr^ : total population, it seems advisable that sL e 
evaluators also prepare statistical report^for^. of the LEAs. At the local level, the detS report 
and the executive summary should be mSnimH; jquirements. usxaiiea report 



Preparing the ContenrHHd Information Packaging 

/I » ^Z"}^'.^^}^ information about vocational programs is prepared for one of three types of functions 
(1) public information or public relations, (2) administrative decision making, and (3) program 
decision making Each of these uses requires a different strategy for content development and style 
of packaging Under normal circumstances r is Tiot advisable to photocopy the computer-generated 
printouts for distribution. The three types o : functions for which ^formation is prepared aT ■ 
packaged are treated separately in the discission which follows. 

Public lhformation/Re!ations 

This function is one of keeping the general public informed aaout the needs and achievements 
of vocational education. The information must be packaged in as simple and efficient a way as 

et°^! ( '979!'r6TsTatr '"'^ '"^ ^'^^'^ ^«'=°"^"i«nded. Starr 

Whether graphic or tabular techniques are used, three factors underlie 
satisfactory display of quantitative data: simplicity, clarity, and effective- 
ness. The graphic and tabular forms of data display must be easily read 
and understood, and must be presented in a manner which will facilitate 
ease of comprehension and retention. These purposes require consideration 
of: (a) the nature of the data; (b) the purpose of the display; (c) the medium 
for presenting the data; and (d) the audiences to whom the data are presented 
One or all of these factors may be pertinent to any situation where data 
are presented or displayed. 

After the visual display is completed, it should be examined carefully with the followina 
questions in mind: ^ 

1. Does it convey the intended message? 

2. Does it display the relationship clearly? 

3. Can it be displayed differently? 

4. Does it have eye appeal? 



97 



although these are basic requirements, more often than not individuals packaging information do 
ot address the foregoing questions, and thus run the high risk of failing to communicate 
-rrectiveiy to the Intended audience. 

-administrative Decision Making 

Many decision makers do not have time to do extensive reading and analysis, because of their 
resaonsibihties and priorities. Therefore, it Is a waste of time and effort.to provide these Individuals 
with extensive sets of unsynthesized data (see table 7), expecting them to do their own analysis of 
Dutcomes as they relate to problems they are trying to solve. Decision making information that is 
nrovided to managers should be synthesized and packaged (see table 8) Into a very practical, usable 
acceptable form. Charts and tables of data need to be clear and concise for Immediate understanding. 

A great deal of responsibility rests on the shoulders of the evaluator or data analyst who 
primarily bear the responsibility for interpreting and packaging the information for dissemination. 

Program Decision Making 

As indicated earlier, at the local level and at the state level, for those concerned with individual 
program areas in development and improvement, the detailed report and executive summary should 
be minimal requirements. Depending on Individual requirements as negotiated by the evaluator and 
the client(s), the raw data and processed information may also be prepared for individual Institutions 
and agencies. 

Care must be taken to maintain conficlentiality of data and information. At a minimum, schools 
should receive all information and data recorded about their own programs as well as summary data 
for the LEA and region as a whole. Consideration may also be given to the grouping of schools by 
certain characteristics such as size, demographics, or socioeconomic characteristics. Care should be 
exercised in sharing the detailed data/Information from individual schools with other schools. 
Attention must be given to federal legislation requirements protecting the confidentiality and orlvacv 
of individuals. ' *^ ^ 

Summary 

I n summary, interpretation and packaging of data and information on students' satisfaction 
with their training and jobs require strict attention to the needs and characteristics of the audiences 
for which the information is being prepared. Table 9 Identifies general factors to be observed in 
organizing and formatting a report, and considerations for the graphic display of data. 
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TABLE 7 

Job Satisfaction Characteristics 
by Degree of Satisfaction of Technical Program 



Characteristic 



Count 
Row Pet 
Co! Pet 
Tot Pet 



Highly Highly Row 

Satisfied Satisfied Dissatisfied Dissatisfied Total 



Salary 



Fringe Benefits 



Supervision and 
Management 



Company Policies 
and Practices 



Working 
Conditions 



Potential for 
Advancement 



1.00 



2.00 



3.00 



4.00 



1.00 



2.00 



3.00 



4.00 



5.00 



6.00 



5 
25.0 
15.6 
04.2 



5 
25.0 
15.6 
4.2. 



3 
15.0 
9.4 
2.5 



4 
20.0 
12.5 

3.3 



7 
0.35 
0.22 
0.06 



8 
0.40 
0.25 
0.07 



7 
35.0 
12.5 

5.8 



8 
40.0 
14.3 

6.7 



J5 
75.0 
26.8 
12.5 



11 
55.0 
19.6 

9.2 



8 
40.0 
14.3 

6.7 



7 

35.0 
12.5 
5.8 



6 
30.0 
28.6 

5.0 



4 

20.0 
19.0 
3.3 



1 

5.0 
4.8 
0.8 



3 
15.0 
14.3 
2.5 



4 
20.0 
19.0 

3.3 



3 
15.0 
14.3 
2.5 



2 
10.0 
18.0 

1.7 



3 
15.0 
27.3 

2.5. 



1 

5.0 
9.1 
0.8 



2 
10.0 
18.2 

1.7 



1 

5.0 
9.1 
0.8 



2 
10.0 
18.2 

1.7 



2.0 
16.7 
2.0 



2.0 
16.7 



2.0 
16.7 



2.0 
16.7 



2.0 
15.7 



2.0 
16.7 



Column 
Total 



32 
26.7 



56 
46.7 



21 

17.5 



11 
9.2 



12.0 
100.0 
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TABLE 8 



^ Degree of Job Satisfaction of 
Former Technical Students. by Characteristic 



Characteristic 



S alary^ 



Fringe V 
Benefits\ 



Supervision and 
Management ' 



Company Policies 
and Practices 



Working * 
Conditions 



Potential for 
Advancement 



> 



L 



rtri 



H.S. 



H.S. 



D. 

H.D. 



H.S.I 



S.' 



H.S. 



D.I 



TTK" 



H.S. 



D.' 



H.DJ 



M.S.I 



IT 



Li 



33 



01 

H.S. - Highly Satisfied 
S. — Satisfied 
D. — Dissatisfied 
H.D. - Highly Dissatisfied 



) 



10' 20' 30 40 50 

Percentage of Respondents 
100 
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TABLE 9 

Factors to Consider in Preparing Reports on 
Former Students' Satisfaction with their Training and Job 



1. Include the survey instruments In all reports and presentations, if possible. 

2. Set up local reports based upon local requirements, but also include information required by state 
and federal mandates that relates to local programs receiving state and federal support. 

3 Break out and analyze data as-much as possible, but not all information and analyses need be ■ 
/ presented to everyone. 

4. Make tabular summaries in the shortest possible form. Oral presentations are best with this type of 
mtorrnation. Follow survey form de.slgnwhen making presentation: question 1, 2, 3, etc. • 

5. Prepare and present a. copy of the report for each member of the audience during oral presentations 
If practical. Be sure to present data in the form and content applicable to the particular audience. ' 

76. Summarized reports are usually the best format for presentation to most groups; however more 
/ detimtive information is required in certain instances. 

7. The comments section of a survey instrument is very important. This area many times reveals needs 
and Shortcomings, especially if a particular comment is repeated several times. 

8. Scan the comments section for the most frequently mentioned items/and make summaries for 
inclusion in reports and presentations. Refrain from using actual names (of teachers, administrators 
etc.) given in comments. . ; ' 

9. Do not attempt to include survey information from a student surveyed in the wrong curriculum area 
hor example, students are sometimes included in vocational surveys who may have taken only one 
vocational course and who are, in fact, pursuing college prep courses. Their inclusion in a vocational 
survey biases the information and increases response error% 

10. Data tabulation and analysis methods must be appropriate, or nonuse of the data is assured. 

11. Break out district data on a per-school or program-per-campus basis. Most administrators are interested 
m information about their institution and their graduates. Compare schools to county data or 
program to program data; but avoid institution-to-institution comparisons. 

12. Break out data by program, where applicable. This format is very meaningfu I for vocational reports 
and presentations. It gains teacher support for the activity. 

13. Compare and present data by courses and programs, not teachers. Many students react to questions 
about courses or programs by nature of their relationship with a teacher, and this tendency biases 
the data. 

14. Keep any sophisticated statistics in a separate section for those who wish to review them but do 

not present statistics throughout the report. Rates of use will go down, especially in audiences with ■ 
little or no background in research methodology. 

15. Percentages, graphs,-and charts are information display methods to which most populations best relate. 

16. The main use of follow-up information, as it currently exists, islto disseminate general indications of 
the condition of an institution, district, etc. Include an abstract! of each report that summarizes these 
findings and gives a brief description of the information gathering/analysis techniques. 

17. Make sure reports appear neat, are printed on good quality paper, and are in readable form. 

Franchak, S. J. and Spirer, J. Guidelines and Practices for Follow-ul Stldies (Columbus Ohio- The 
^National Center for Research in Vocational Education, The Ohio State University, 1979)', p. 127. " 
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GLOSSARY 



Case Study Method. A case study is an intensive/detailed analysis and description of a single 
organism, institution, or phenomenon in the context of its environment. The term has long 
been associated with law, medicine, and social work. The case study is also considered a 
legitimate method of inquiry in the social sciences and frequently is extended to several cases ' 
at once (Anderson eta I. 1975, p. 46). 

Data Preparation. The data collected in an evaluation have to be organized to make them readily 
accessible and usable. When the evaluation is small in scale (few subjects, few measures simple 
design), the problems of data preparation are usually not great. However, as the size of the 
evaluation increases, so do the problems of data preparation. Whether data are hand-scored 
keypunched, or^optically scanned, they first need to be prepared to ensure good-quality data 
sets^and efficient data analyses. Data preparationf includes all steps taken to anticipate the 
needs ^ anyone working with data at some future time (Anderson et al. 1975, p. 112). 

Descriptive Statistics. It is a field of statistics which is concerned primarily with summarizing data 
systematically for ease of comprehension (Anderson et al. 1 975, p. 400). 

Inferential Statistics. It is a f iefd of statistics which is concerned with usinp data as the basis for 
mak,ing certain generalizations and interpretations (Anderson et al. 1975, p. 400). 

Interviewing It is a common form of collecting data employed in survey research. Data are 
collected by asking questions of people. 

Multivariate Analysis. It refers to a series of statistical techniquk used for analyzing a set of 

outcome variables observed on a number of students or experimental subjects. The dependent 
variables are analyzed with reference to one cr rribre independent variables (Anderson.et al 
1975, p; 250). ■ 

Quasi-Experimental Design. To paraphrase Campbell and Stanley, there are many education and 
training situations in which the evaluator can imroduce something like experimental design 
but where he lacks control over when students are exposed to the program or which students 
are ex.oosed to it, including the inability to assign students at random to experimentaf and 
control groups. Collectively, such situations may require the use of compromise procedures 
called quasi-experrmental designs (Anderson etal. 1975,11, 301). ^ 

Questionnaires. A questionnaire is a group of printed questions used to elicit information from 
■ respondents by means of self-report. The questions may be open-ended, requiring respondents 
to select onie or more answers from among thbse provided. The respondents may also be 
provi;ded with checklists or rating scales. Questions may be concerned with the respondent's 
personal background, factual knowledge, or attitudes and opinions (Anderson et al. 1975 
p. 311). 
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Special Needs. Persons who meet the requirements under the law for one of the categories (handi- 
capped, of limited English-speaking ability, disadvantaged [economic and academic] ) whether 
or not special services are provided by vocational education per se (Franchak and Spirer 1978, 
p. 159), 

Univariate Procedures, It refers to a series of statistical techniques used for analyzing a dependent 
variable, regardless of the number of independent variables. 

i 

Unobtrusive Techniques. Data collection techniques that do not require contact with individuals- 
observation, physical trace, and archives or records (Kester 1975, p. 53). 
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ANNOTATED BIBLIOGRAPHY 



Job Satisfaction 
■ /' ■ • ' 

Anthony, W. P. "The Study of the Effectiveness of Public Postsecondary Vocational -Technical 
Education in Preparing Graduates for the Labor Force." Ph.D. dissertation The Ohio State 
University, 1971. 

Purpose of the study was to determine the extent to which public postsecondary vocational-technical 
education leads to success in the labor market. Data were collected from graduates of the Columbus 
Ohio, Technical Institute (CTI), and Columbus, Ohio, Public High Schpol graduates. Job satisfaction 
for all jobs and present job was assessed by asking the graduates to respond to one question "On the 
whole, were you satisfied with the work?" 1 -very satisfied, 2-satisf led, 3-dissatisf led, 4-very 
dissatisfied. Findings included: (1) There is not a significant difference between the job satisfaction 
of the CTI graduates and job satisfaction of the high school graduates. An overall conclusion reached 
by. the author was: "CTI graduates were more satisfied with their present jobs and were more likely 
to remain with each employer for a longer period of time." 

Ballo, Gary R. "A Comparative Follow-Up Study of Graduates and Non-graduates from the Voca- 
. tlonal Division of the Lewis-Clark Nprmal School." Lewiston, ID: 1971, Lewis-Clark Normal 
School. 

This study sought to compare graduates and nongraduates of the vocational division of, Lewis-Clark 
, Normal School. The sample consisted of students who attended vocational programs at LCNS, from 
1966-1970. Graduates were those who received a certificate or degree, nongraduates were identified 
from program and school records. Job satisfaction was operationalized by having students indicate 
the extent to which they ".liked" their present occupation and present job. There was no significant 
difference between. graduates and nongraduates with respect to job satisfaction. 

Bouchard, T. J., Jr. "Field Research Methods: Interviewing, Questionnaires, Participant Observation, 
Systematic Observation , Unobtrusive Measures. " I n Handbook of Industrial and Organizational 
Psychology. Edited by Maryin D. Dunnette. Chicago, I L: Rand McNally College Publishing 
Company, 1976. / 

In this chapter, the special characteristics of field settings and the difficulties that researchiin such 
settings entails were reviewed. When compared to research in laboratory settings, it was cp^hcluded 
that for rhany purposes field studies could generate extremely useful, practical, and theoretical 
knowledge. Five major field methods were descr'ibed in considerable detail. None of the r/iethods 
were evaluated relative to each other, however, because the author believes that there is Jio single 
best method, in any absolute sense. ^\^^ 

' > . ■ ^- ■ r 

Burgess, Dale E., ed., Michigan Department of Education, Vocational-Technical Education Service 
Follow-up Survey of 1978 Graduates for Warren Consolidated Schools. Warren, Ml: Warren 
Consolidated School District, 1978. ' 



This study determined the relationship between secondary education programs and employment, job 
satisfaction, earnings, and/or further education. Another purpose was to obtain graduates' evaluation 
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of theinraining and information for planning and deveiopmerit^AII 1978 secondary graduates frnnr 
the Warren Consolidated School district werercontacted by mail or telephone. Job satisfaction war 
measurH2-by aslcinrg participants how satisfied they were overall in their present job. 47 percent of- 
vocational graduates reported that they were "very satisfied" with their jobs compared to 37 pert^ 
among Tonvocatlonal graduates. 

Casey, F. M., ed., Work Attitudes and Work Experience: The Impact of Attitudes on Behavior 

Washington, D.C.: U.S. Department of Labor, 1979. 
This monograph describes the findings of a special exploration of data from the National Longitudinal 
Survey (NLS), a 1 5-year study of the attitudes and work experience of four age-sex groups: men 
aged 45 to 59, women aged 30 to 44, and young men and women aged 14 to 24 years at the date of 
the mitial interviews. Jhis special effort further subdivided these four groups by race and examined 
the interaction of work-related attitudes and subsequent behavior for each of the eight resulting 
age-sex-race groups. 

The findings reported here represent an important step forward in research concerning worker 
motivation, since they make it clear that attitudes do influence subsequent work behavior. Specifically 
It was established that individuals who felt they could Influence their future through their own efforts ' 
later experienced greater success in the labor market than those who were less optimistic. Similarly 
the relationship between job dissatisfaction and turnover was clearly marked in each of the eight 
age-sex-race groups. The evidence also indicates that dissatisfaction results in extensive costs borne 
by workers in terms of increased unemployment, decreased labor force participation, below-average 
growth in annual earnings, and a low^r rate of promotion; For women in particular, their degree of 
commitment to work, their attitudes toward the propriety of mothers' working, and their hgsbands' 
attitudes toward wives' labor force participation all bpre a measureable relationship to subsequent ' 
work experiences. ' ' , 

Conroy, William G., Jr.; Diamond, Daniel E. 'The impact of Secondary School Occupational 

Education in Massachusetts." Lowell University, MA: College of Management Science, 1976. 
The study exarnined the impact of various secondary school programs, upon the lifestyle of students. 
Primary attention was focused upon school-labor market relations. A probability sample was drawn 
from students who had been enrolled in secdhdary occupational and nonoccupational programs in 
Massachusetts. Job satisfaction was one of the outcome variables examined and was operatiohalized 
by a "very satisfied; satisfied; not satisfied" questionnaire item. Occupational students proved to 
have slightly higher job satisfaction than nonoccupational students. 

Dawis, R. V. "The Work Adjustment Project."-|n Programmatic Research and Development in 

Education. Edited by F. C. Pratzner and J. P. Walker. Columbus, OH: The Center for Vocational 
and Technical Education, 1972. 

Dawis described the continuing programmatic series of research studies on the general problem of 
adjustment to work. The project was characterized by a "circular" model of research and development 
and illustrates an overall "linear" progression through three stages of activity. 

The circular model of research and development on work adjustment involves the interaction and 
interplay of three components: theory, methodology, and data. Some of the developmental impli- 
cations which resulted from activity on each of the components were illustrated by reference to the 
Work Adjustment Project. 

The Work Adjustment Project was further described as .having moved linearly through three stages 
of development: an exploratory stage where guidelines for researdn and development are established 
and the boundaries of the problem area are described; a descriptive state where the problem area is 
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Several alter^ive directions for future research and development were identified and briefly discussed 

advancem^t were named most often both as satisfiers or diss^tis'fSrs P°ss'b'l't.es for 

Gruneberg, M. M. Understandmg Job Satisfaction. New York, NY: John Wiley and Sons 1979 
conZrTZ'nt''V satisfaction. Chapter 1 introduces the topic and its historical 

ihe ' examines w'ays of improvi^g^S^tr^ 

"''"ihf clmiihu;in^^^^^^^ ^'.'"^ ^"^'■"^ "The Transition from School to Work- 

Northe°a~ 

s™c^2iul iKor'^^^^^^ preparing students for a " 

successtui labor market role. It was divided into two parts focusing on the graduatinq classes of 1 9B6 
and 19,2. Samples were drawn from nine Boston area high schools and inckjded^ Ks from ^ ^^^^ 
cooperative vocational, education.programs, regular vocational education programs and 
academic programs. _Job satisfaction was measured by respondents indicating 'i^S Si fled 
or unsure as to their first job and employer. The finding was that graduates ofiheToo;^^^^^^^^^^ 
programs were significantly more satisfied with their jobs than were graduatero the other p oqrams 

operative .tud^ts having more 

■ ' ' ■) ■ ■ ■■ . . ■ . ... ■ ■ 

Katz Douglas S. ; Morgan, Robert L.; and Drewes, D. W. "Vocational Education and Urban Youth- 
A Fpl ow-Up Study of 1968, .1971 and 1972 Graduates of the Public School of the SiSict 
of Columbia." Raleigh, N€: Center for Occupational Education, North CaroHna sL Ser^ . 

The stucy aimed to specify the design for a foitow-up system for the District of Columbia Public 
School .system. Data were collected by teleptane survey. The deliverative/purpos°ve Smple w^^^^ 
dravvj, THom graduates of Washington D.C. pufenic vocational and academic high schoorML^Ss 
whittfi?"'°" ,'J'='",d^,^'°"'^tjsf led emp^yed vocational graduates were with the rjol^ 
f^iPHH q ntJ- """"^ I ^9''"' ^^^^^^^ *hey would recommend their job t^ a 

Stheir'inr^n^S^ ninety percent of vocational graduates were Satisfied 

with tbe.r jobs, and (2) sixty percent said they would strongly recommend their job to a friend 
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general education or a skill training progrTm'joS SSnn .^T'' ''"'""^'^ ^^^'^"^^ ^° ^^'^her a 

of iob satisfaction is tlien analyzed and Sau?sh.rf ^T'l" advocates. Tlie concept 

affecting iob attitudes are ai^l^6 ti Sr7^^^^ After basic job dimensions 

critically analyzed. Major findings concerZc h^IS ,h! ™"l="t theories of pb satisfaction'-are 

rfXSb--— d~^^^ 

."adSrru'^'tirnrX"'^^^^^^^^^ °' r -^-"^^ 

;;;'^^=v:^::»fe-i-^^^ 

Srs=,'^;;^^^,"s3lSo'n°S^^^^^^ -plovabHity, job ' 

^^Csl^rltt^dTntX 

responding in a Likert scale to twen y f iv^iT^iw^^^^^ operationalized by students 

'■my job is boring". Findings were tl^at i^b StiS»«if„ l ^ questions or statements such as 
vocational graduates and otheTrraCes or^^^lT . J'ofJWi'icam difference between . 

•heir jobs, and are less inclined ^o cZg^'j^blir t^^rSS ^,1:^27 ' '"'^^ '^"'^ 

s™ raSrsitVsfying^ 5— 'he chandes o, 

education and job sitisfaaion Source of infLm^ • PjV^hological processes that may link 

published research reportsV^r^g-r'dlrt^rc^'Lriro^e'Z^^^^^^^ 
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and job satisfaction and seconder/ analyses of nine national surveys designed for purposes other 
.than understanding the relationship between education and job.satisfaction. 
Five of the sixteen studies found a positive association between education and job satisfaction three 
found a negative association, and eight reported the relationship to be either nonexistent or equivocal. 
Analyses of the nine national surveys found no increment in job satisfaction with succeeding years 
of education. However, while jip relationship was found between education level and job satisfaction 
among workers who had not gone to college, those who had obtained college degrees were consis- 
tently more satisfied with their jobs than were employees without degrees. All but one of the surveys 
identified "credentials effect," evidence that there was no payoff in job satisfaction from having 
college training unless one alsa received a college degree. Level of education waslignificantly and 
positively related to overall quality of employment, and the greatest increment in quality of employ- 
. ment occurred.at those points where educational credentials were conferred. 
The report concludes with several recommendations for further research and for policy changes on 
the part of employers and educators. Among the recommendations for policy changes are the 
following: 

' • Employers and educators should be aware of the occupational needs of the over-educated. 

• The educational requirements established for jobs should be reexamined. 

• Job design, where possible, should take into account the increasing education level of the 
labor force. 

• Educators should place greater emphasis on general skills, anticipating ihe many job changes 
in life. 

• Training for specific jobs should be reserved until it becomes necessary-fcr the worker to 
receive such training. 

• Educators should not justify "every unpleasant thing in school" as essential for securing a 
good job. 

Quinn, R. P., and Stanes, G. L. The 1977 Quality of Employment Survey. Ann Arbor, M!- Institute 
for Social Research, 1979, 

This volume contains descriptive statistics on the 1977 quality of employment survey with comparison 
data from the 1969-70 and the 1972-73 surveys. The measurement of job satisfaction was approached 
m two ways: first, with a generaj set of facet-free questions phrased so that workers can invoke any 
considerations of their choice; second, with a series of questions about specif ic facets of the worker's 
job and employment conditions (e.g., pay, hours, etc.).. Responses to the general questions were 
averaged to form an index called "Facet-free Job Satisfaction/' The specific responses were clustered 
bV topical (and statistidal) similarity, with indexes for each topic and also one for all topics com- 
bii^ed ("Facet-specific Job Satisfaction"). The topical areas are: Comfort, Challenge, Financial 
Regards, Relations with Coworkers, Resource Adequacy, and Promotions. With the addition of an 
overall index ("Overall Job Satisfaction") combining Facet.free Job Satisfaction and Facet-stf^cific 
Job\Satisfactiori. 

Richardson, William B. "An Analysis of Factors Influencing the Earnings of Incwna High .School 
Vocational Graduates." Indianapolis, IN: Indiana State Department of Purnc Instruction 
Division of Vocational Education, 1975. 

The puVpose of the study was to quantify the relationships between earnincsrof imdiana vocational 
graduates and selected variables hypothesized to affect those earnings. Job sEtisjaction was opera- 
tionali.zed with an item asking respondents to indicate their feelings toward.dier-present job as 
"Highly like," "Moderately like," "Indifferent," or "Dislike." Conclusions werHthat persons who 
like their present jobs earn significantly more than those who are dissatisfiectEnd that student 
satisfaction with training does little to explain earnings differentials. 
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Weiss, D. J. "Multivariate Procedures", m Handbook of Industrial and Organizational Psychology 

Edited by Marvin D. Dunnette. Chicago, IL: Rand McNally College Publishing Company, 1976. 
Methods of multivariate covariation analysis (factoraf^f^lysis, cluster analysis) are classified and 
explamed m nonmathematical terms, within the co~ - of research in industrial-organixational 
psychology. Classification schemes used are designee w ^ -he researcher in choosing methods 
most appropriate for a given applied problem. Decir, , ^ in choosing among multivariate 

prediction models are based on whether the criteria is single or multiple and whether it is 

contmuous or categorical, and whether the research ■ ng t : assume linear relationships between 

the set of multiple predictors and the criterion variat ach method is discussed in terms of its - 
rationale, interpretations, and limitations. Methods v yzing matrices of multivariate correla- 
tional data are divided into those methods that analv/u al variance (components analysis and 
cluster analysis) and methods for analyzing common variance (factor analysis). Within each of these 
major classifications additional distinctions are made among available techniques to help researchers 
Identify the methods most appropriate for a given problem and to assist in the interpretation of the 
-results of analyses using the methods. 

Von Stroh, Gordon E. "A Socio-Economic Study of Vocational-Technical Education Students " 
Oklahoma: University of Oklahoma, 1968. 

This study examined a number of vocational education outcome variables, among them job satis- 
faction and student satisfaction with training. The sample was.diawn from graduates and dropouts 
from a single vocationahtechnical school in Oklahoma. Data were collected by questionnaire Job 
satisfaction was operationalized by a single categorical item, "Satisfied/Not Satisfied." it was found 
that 70 percent of the respondents were satisfied with their job. • 



Student Satisfacaisn with Training 

Anthony, W. P. "The Study of Effectiveness of Public Postsecondary Vocational-Technical Education 
in Preparing Graduates for the Labor Forca." Ph.D. dissertation. The Ohio State University ' 
1971 . „ . 

The study determined. the extent to which pubEic postsecondary vocational-technical education leads 
to success in the labor market. Data were collected from graduates of the Columbus Technical 
instmite, and Columbus, Ohio Public High School graduates. Stlident satisfaction with training was 
operHtionalized by having respondents rate various aspects of their training such as instruction, 
racil-ties, counseling, etc., on a four-point scale. Overall ratings given by CTI graduates were slightly 
higherthan those given by high school graduates. 

Ballc;, Garv R. "A Comparative Follovy-up Study of Graduates and Non-graduates from the Vocational 

DivisiDn of the Lewis-Clark Normal School". Lewiston; ID::.Lewis-Clark Normal School, 1971. 
Thisstudy compared graduates and non-graduates of the vocational division of Lewis-Clark Normal 
SchDDl. The ample consisted of students who attended vocational programs at LCNS from 1966-1970. 
GracmatBs were those who received a certificate or degree, nongraduates were identified from program 
andrscfaooii records. Student satisfaction in training was operationalized by students indicating 
"excellent, good, satisfactory, or poor" in appraisal of their programs' instructional and counseling 
components. Students also Responded to questions regarding how well the program prepared them 
for their first job, and whether they felt th6y received their money's worth. There was no significant 
difference between graduates and nongraduates in their evaluation of instruction and counseling. 
Over SO percent of both groups felt that they received their money's worth and that the program 
prepared them well for their first job. 
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Burgess, Dale E., ed., Michigan Department of Education, Vocational-Technicai Education Service 
Follow-up Survey of 1978 Qraduates of Warren Consolidated Schools Warren Ml - Warren 
Consolidated School District, 1978. 

This study determined the relationship between secondary education programs and employment 
job satisfaction, earnings, and/or furttner education. Another puraose was to obtain graduates' ' 
evaluations of their training and information for planning and development. All 1978 secondary 
graduates from the Warren Consolidated School District were contacted by mail or telephone 
Students satisfaction with training «as measured by participants responding to the question "How 
weN did your courses prepare you tD:Jo what you are doing now?". Sixty-six percent of vocational 
g adua tes were very satisfied" withitheir training compared to 60 percent among nonvocational 
yr duUstBS. 

Career and Vocational Education Section, Oregon Department of Education, "Follow-up of 1977 
of'I'du^tioM^^^ Program. Graduates and Early Leavers". Salem, OR: Oregon Department 

The study as^ssed the relatedness of school programs to students' job and how well school experiences 
JnTpirlvf n iob^ SiLdent satisfaction with training was also assessed. All g?aS? 

tniSt^ f " fi° O'-^S"" ^.h'9h school class of 1977 were surveyed by mail. Students were asked 
Itnlnt ''.T"* was related to the training they had received. They also evaluated 

T^SlT'T' ^ P''^P^.''?'°"' preparation for further education. A majority of respondents 

rated their job preparation as either satisfactory or excellent. The most significant finding was that 
students WHO took more vocational courses had jobs that were hiore related to their training. 

Feltsenhausen Joyce L, et al "Follow-Up Report on Illinois 'Class of '71 ' Occupational Program 

Alumni, Final Report. Eastern Illinois University, Charleston, I L: Center for Educational 
otudies. 

The stated objectives of the study were to develop a follow-up data system and to test the system by 
conducting an extensive follow-up stsidy of the relatedness of occupational programs and labor 
market experiences Respondents included all occupational program graduates from 20 percent of 
Mlinois schools in 1971 and their employers. Student satisfaction with trainingwas operationalized 
mp22 tl'dln'r*;;- ^ ^^f P^^IT'^^^ °\ ^^eir training in preparing them for employment. These assess- 
ments tended to be very low. The only area in which the ratings were high was use of tools and 
equipment. 

Gammel, J-anne M.; Brodsky, Stanley; and Alfred, Richard.-"Follow-up Study of 1969-1 975 

Gradijfatesof-the Division of Technology of New York City Community Colleges." Biiooklyn . 
NY: iMew York City Community Colleges, 1976. . ' 

The purpose of the study was to examine the attitudes and postcollege experiences of technology 7 
graduatesat New York included in the study and were surveyed by mail. Student satisfaction witV 
training v^s measured by graduates indicating how helpful their training had been in getting their/ 
first job, how helpful their training was in performing:their first job, and whether or not they woiid 
recommend their program to a friend., Findings varied among prognam areas but over 50 percent/in 
all areas f6lt their program had been helpful in getting and performing their first job. Electrical / 
technology graduates had higher than average perception's of the value of their training while archi- 
tectural technology graduates gave a lower evaluation. Over 80 percent of graduates in all areas 
said triey would recommend their program to a' friend. 
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Hall, Wilbur; Gray, Rodney; and Bsrry, Arthur. "Follow-up Study of Machine Tool Technology and 
Building Construction Graduates." South Portland; Maine: Southern Maine Vocational 
Technicai Institute, 1975. 

The purpose of this study was to assess the labor market experiences of building construction and 
i;j^;'"^i°°'^!'=^;"°'°9y graduates of Southern Maine Vocational Technical Institute from 1970 
ia/^ and 1974. Information for program evaluation was also solicited from graduates. All qraduates 
from the above years were surveyed by mail or interviewed. Respondents indicated their overall 
satisfaction with training and also their satisfaction with instruction and training equipment A 
majority of graduates rated their training as excellent or very good..A majority felt that the equip- 
ment used during training was similar to that used on the job and 74 percent found it easy to adapt 
to industrial equipment. Instructors were rated as very knowledgeable by over 90 percent of all 
graduates. 

Hanby, Jeanette; Harper, Ronald; and Myers, Leo. "A Comparison Study of the Benefits of Secondary 
and Postsecondary Vocational Education." Portland, OR: Northwest Regional Education 
Laboratory, 1978. ^ 

The purpose of the study was to assess graduates' perception of their training, employers' assessment 
ot graduates training, and to conduct a comparison of perceptions of quality of life. The student 
sample was selected fi-om Montana High School graduates who had been employed in Montana at 
least two years. Employers who often hire vocational education graduates were selected to represent 
various sizes and types of businesses. Data were gathered through telephone surveys and mail question- 
naires. Student satisfaction with training was operationalized by asking students to rate their voca- 
tional programs as "excellent, good, fair, or poor." Ratings for postsecondary progr-ams were highest 
vvith ratings for secondary vocational education programs and general academic programs being about 
the same. However, more'postsecondary graduates than secondary vocational graduates'expressed a 
need for more experience during training. 

^ ■■ ■•■ . i 

Hernstadt Irwin L.; Horowitz, Morris A.; and Sum, Andrew M. "The Transition from School to 
Work: The Contributions of Cooperative Education Programs at the Secondary Level " Boston 
MA: Northeastern University, 1979. ' 

This study tested the efficacy of different secondary school programs in preparing students for a 
successful labor market role. The study was divided into two parts focusing on the Graduating classes " 
ot I9bband 1972. Samples were drawn from nine; Bbston-area high schools and included students 
from cooperative vocational education programs, regular vocational education programs and general 
academic programs. During a final interview students were hsked to assess the effectiveness of their 
high school programs. Significantly more cooperative and regular vocational education students felt 
they had been "very well" or "well" prepared for work than did general academic graduates More 
than 50 percent of the genersi .academic graduates feltrthey had been "unprepared" or "verv ooorlv 
prepared" for work. 

Hodges, Stanley L.."Emplovee-Employer Assessment of the Effectiveness of Agricultural Mechanics 

Training Received at Modesto Junior College." Unpublished dissertation, 1973. 
The stated purpose of the study was to gather information regarding the quality of job preparation 
offered by the Agricultural Mechanics program at Modesto Junior College from the perspectives of 
former students and their employers. All Agricultural Mechanics majoi-s who graduated froin Modesto 
Junior College between 1965 and 1972 and their employers were surveyed by mail. Student satis- 
faction with training was operationalized by having students indicate where they felt they learned 
most about their skill. Choices were high school, Modesto Junior College program, apprenticeship 
program, on-the-job, and elsewhere. In all nine skill areas the majority of former students said they 
felt they learned more at Modesto Junior College. 
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